Custom Carrier & Headless operation of Carriers

Adding your own carrier / lastmile solution to fulfillmenttools

Apart from the provided carriers fulfillmenttools offers another option of carrier usage: The so called Custom Carrier. Those represent carriers that are not natively supported by fulfillmenttools, but are connected by an external service or operative process.

The concept is that a an own last mile fleet or an arbitrary carrier is integrated seamless into fulfillmenttools. Therefore a mechanism is implemented, so that partners or customers can do this work themselves, stay nimble and support their innovations in the best way possible.

Some examples for a custom carrier setup could be

• Sending parcels via a local bike carrier, that is called by phone or messenger when needed

• the shop owner, who takes parcels to a local carrier pickup point whenever it suits the operational process (maybe on his way home after work)

• an own carrier fleet that delivers goods on its own schedule using detached tooling (like a standalone routing service & delivery app)

Multiple custom carriers possible While it is not possible to have more than one provided carrier in fulfillmenttools, it is very well possible to configure multiple custom carriers. Since the carrier key is used for identification and thus must be unique, a custom carrier must have the key starting with CUSTOM_ followed by any appendix (e.g. CUSTOM_BOSSONBIKE) to ease identification.

Usecase: Headless connection of a carrier to fulfillmentools

In combination with a custom carrier it is possible to connect an own carrier by yourself to the fulfillmenttools platform. Let's say you want to ship goods using a carrier that is not provided within fulfillmenttools. Within this tutorial we will take a look at a possible solution that show cases the integration of an alien carrier with fulfillmenttools, the implemented processes and used clients.

The overall idea is, that fulfillmenttools allows for easy headless integration of carriers by using eventing and API interactions. This enables an external service (implemented & operated by a partner or a customer) to harness all the needed complexity to successfully get a certain label from an arbitrary carrier.

Concept

Please see the following diagram and the explanation below to further describe the overall approach here:

Concept & used components for this showcase

1. A subscription is needed at fulfillmenttools, that triggers the whole flow It depends on the process of the customer which explicit Event is used as a trigger, but it would make sense to react to an Event close to the finishing of the packing step (e.g. in this example PACKJOB_CREATED). This Event is send to the external service that handles the actual label request from the Carrier API.

2. External Service has logic to take some decisions & gather needed data The external service (operated by a partner or the customer) receives the event and uses it to take some decisions, for example which exact carrier or carrier product should be used (based on data that is available to the service), how much insurance value needs to be requested, etc. The data could be provided by services, from a database or even from fulfillmenttools when needed.

3. Gathered data is used to create the Label request towards the actual Carrier API Similar to the mechanism within fulfillmenttools platform the external service will now use the gathered data to request a label & negotiate a pickup with the carrier. The label itself and possibly other data (like pickup dates, transition times, etc.) is then received from the carrier during this process.

4. Carrier Label & Data is stored at fulfillmenttools Once the label, track and trace information and possibly additional data is received by the external service this data should be supplied to fulfillmenttools to allow for seamless integration with the process. The label for example would be uploaded as part of a documentset, received track and trace data will be stored as a result at the parcel entity, etc.

5. Setting Handover-States based on T&T Data A feature of fulfillmenttools is to also set the handover of shipped parcels to “handed over”. This is again something the external service can do: It needs to connect to the carriers T&T API and decides for itself when the handover in fulfillmenttools is set to “HANDED_OVER” - depending on the desired process.

Proof of Concept: Connecting Uber direct to fulfillmenttools

Overview

As a proof of concept we will show how to connect a the carrier Uber direct to fulfillmenttools. Therefore we will create a Service that we call Uber Connector Service.

Please note that this is a mere Proof of Concept which means: Uber direct is chosen because of the easy technical connection & usage to the service. It can be substituted with any other carrier if needed.

For this POC we decided for a lot of concrete parameters like which Event to use, which data is needed, when the handover takes place, etc.. Please understand: This is not the only option but merely one road of many that lead to rome. The adaption to other events as triggers of a more complex logic could be applied additionally if needed.

In this example the Parcel (with the label pdf) will be automatically ordered, when a PackJob is created for a given Process. The following diagram illustrates the used events & components:

Necessary steps

We decided to connect to Uber Direct as a carrier: They offer a simple API to request pickups & deliveries with little effort.

The external service itself is hosted within a Google Cloud Project as a simple CloudRun service which scales to 0 and reacts to http requests coming from a fulfillmenttools subscription.

Prepare Service Infrastructure in GCP+

Since we decided to write this proof of concept in kotlin we followed this Guide in order to create a simple running service using kotlin as programming language: Quickstart: Deploy a Kotlin service to Cloud Run | Cloud Run Documentation | Google Cloud

“Why GCP? What about Azure, AWS or even my Raspberry Pi under the desk?”

We knew this question would come! From an architectural point of view it is irrelevant which platform you use to host & operate your service. You are free in your choice here.

After finishing this step we have a live Cloud Run service deployed that is reachable via Internet without any authentication to its endpoints.

POC != MVP - especially in terms of security!

As this is a proof of concept we neglected security and other complexity to some extend. In this particular case everybody that know the address of this short lived service could call it and create a (test) delivery using our service.

When creating a productive service it goes without saying that reasonable security measures for authentication and authorization have to be provided!

Configuration of fulfillmenttools

Following this provided tutorial we create the custom carrier “Uber Direct” as shown below:

curl --location 'https://your.api.fulfillmenttools.com/api/carriers' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: ••••••' \
    --data '{
      "key": "CUSTOM_UBER",
      "name": "Uber Direct",
      "status": "ACTIVE"
    }'
< 201 Created
{
    "id": "26de36d7-883d-456a-91ef-ed9526cd5835",
    "key": "CUSTOM_UBER",
    "name": "Uber Direct",
    "status": "ACTIVE",
    ...
}

Also we will connect this carrier to the facility where we want to provide this service:

# 469adf78-5aeb-425c-b2dc-91027e69336a being the ID of the facility in this case
curl --location 'https://your.api.fulfillmenttools.com/api/facilities/469adf78-5aeb-425c-b2dc-91027e69336a/carriers/26de36d7-883d-456a-91ef-ed9526cd5835' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: ••••••' \
    --data '{
        "status": "ACTIVE",
        "name": "CUSTOM_UBER",
        "configuration": {
            "key": "CUSTOM_UBER"
        }
    }'
    
< 201 Created
{
    "id": "469adf78-5aeb-425c-b2dc-91027e69336a_26de36d7-883d-456a-91ef-ed9526cd5835",
    "version": 1,
    "name": "CUSTOM_UBER",
    "status": "ACTIVE",
    "carrierRef": "26de36d7-883d-456a-91ef-ed9526cd5835",
    "key": "CUSTOM_UBER",
    "deliveryType": "DELIVERY",
    "facilityRef": "469adf78-5aeb-425c-b2dc-91027e69336a",
    "configuration": {
        "key": "CUSTOM_UBER"
    },
    ...
}

Write a “Uber Carrier Connector Service” to handle the trigger event from fulfillmenttools

All roads lead to rome…

…but we chose the easy route by offering a standalone service that does not need a database or any other components besides the container engine it is deployed to. Hence we needed to take some practical decisions when it came to which event to use & how much is done during this event.

We now extend the created service by adding the following functionality. We only describe subsets of the implementation here.

Implementing needed functionality towards Uber Direct API

First we need to create an account at Uber that is allowed to operate against their Uber Direct API. We obtained the following needed data:

• a Customer ID (UBER_CUSTOMER_ID)

• a valid Client ID (UBER_CLIENT_ID)

• a matching valud client secret (UBER_CLIENT_SECRET)

For the sake of simplicity we will generate a new Bearer Token for Uber API (UBER_TOKEN, valid for 30 days) via curl an provide it as a configuration to our service. We use this call to obtain the token:

> curl --location 'https://login.uber.com/oauth/v2/token' \
  --form 'client_id="UBER_CLIENT_ID"' \
  --form 'client_secret="UBER_CLIENT_SECRET"' \
  --form 'grant_type="client_credentials"' \
  --form 'scope="eats.deliveries direct.organizations"'
< 200 OK
{
    "access_token": "UBER_TOKEN",
    "token_type": "Bearer",
    "expires_in": 2592000,
    "scope": "eats.deliveries direct.organizations"
}

Next in line for our process: We need to implement a call towards Uber for creation of a delivery (Pickup at location A, dropoff at location B). In order to accomplish that we create a new Uber API object plus the implementation of said endpoint to create a delivery:

Apis.kt

class Api {
    companion object Factory {
        fun createUberApi():UberApi = Retrofit.Builder()
                .addConverterFactory(GsonConverterFactory.create())
                .baseUrl("https://api.uber.com/v1/customers/UBER_CUSTOMER_ID/")
                .build().create(UberApi::class.java)
    }
}

UberApi.kt

interface UberApi {
    @Headers(
        "Authorization: Bearer UBER_TOKEN",
    )
    @POST("deliveries")
    fun createDelivery(@Body createDelivery: CreateDelivery): Call<CreatedDelivery?>
}

In addition to those two classes we need model classes for the needed request and response bodies, a repository and a service for better structure and readability of the code.

Implement needed connections towards fulfillmenttools REST API

Furthermore we need to write functionality towards fulfillmenttools in order to create parcels & set results to it. The following endpoints are the most important ones:

FFTApi.kt

interface FFTApi {
    @POST("shipments/{shipmentId}/parcels")
    fun createParcel(@Path("shipmentId") shipmentId: String, @Body parcelForCreation: ParcelForCreation): Call<Parcel>
    
    @GET("shipments")
    fun getShipments(@Query("pickJobRef") pickJobId: String): Call<Shipments>
    
    @PATCH("parcels/{parcelId}")
    fun patchParcel(@Path("parcelId") parcelId: String, @Body parcelPatch: ParcelPatch): Call<Parcel>
    
    @GET("handoverjobs")
    fun getHandoverJobs(@Query("shipmentRef") shipmentRef: String): Call<HandoverJobs>
    
    @POST("documentsets/{documentSetId}/documents")
    fun createDocument(@Path("documentSetId") documentSetId: String, @Body documentForCreation: DocumentForCreation): Call<Document>
    ...
}

In addition to create a parcel in fultillmenttools containing the result of the uber delivery, we also need the ability to set the Handover created in the process to HANDED_OVER:

interface FFTApi {
   ... 
    @PATCH("handoverjobs/{handoverJobId}")
    fun patchHandoverJob(@Path("handoverJobId") handoverJobId: String, @Body handoverJobPatch: HandoverJobPatch): Call<HandoverJob>
}

Provide endpoints for events originating from fulfillmenttools and Uber

Now that we have the functionality itself in place we can define the web endpoint which receives events from fulfilmenttools and Uber. These events will then in turn trigger either the creation of the delivery by an event from fulfillmenttools or (later in the process) the processing of the track & trace events from Uber:

@SpringBootApplication
@RestController
class Application {
    private val fftService: FulfillmenttoolsService = FulfillmenttoolsService() //the service to connect to fulfillmenttools
    private val uberService: UberService = UberService() //the service to connect to Uber direct
    private val pdfService: PdfService = PdfService() //a service that is able to create a PDF as base64
    
    /**
     * Callback for fulfillmenttools when a PackJob was created.
     */
    @PostMapping("/requestPickupCallback")
    suspend fun requestPickupCallback(@RequestBody packJobCreatedEvent: demo.fft.model.PackJobCreatedEvent) = run {
        //Authenticate @ fulfillmenttools
        if (!fftService.authenticate()) {
            return@run
        }

        val shipments: List<Shipment>? = fftService.getShipmentsForPickjob(pickJobId = packJobCreatedEvent.payload.pickJobRef)
        if (shipments.isNullOrEmpty()) {
            logger.info { "No shipment found for Pickjob ${packJobCreatedEvent.payload.pickJobRef} - Aborting." }
            return@run
        }

        val shipment: Shipment = shipments.first()
        if (packJobCreatedEvent.event == "PACK_JOB_CREATED" &&
            shipment.sourceAddress != null &&
            shipment.targetAddress != null &&
            shipment.sourceAddress.phoneNumbers.isNotEmpty() &&
            shipment.targetAddress.phoneNumbers.isNotEmpty()
        ) {
            logger.info { "Processing PACK_JOB_CREATED Event from fft..." }

            val parcel: Parcel = fftService.createProcessingParcel(shipment) ?: return@run

            val sourceAddress = shipment.sourceAddress
            val targetAddress = shipment.targetAddress

            //the UberService implementation puts shipment & parcelId to the created delivery at Uber for later use
            val createdDelivery : CreatedDelivery? = uberService.createDelivery(
                ...
            )

            if (createdDelivery == null) {
                logger.error { "Failed to create delivery." }
                return@run
            }

            val labelBase64 = pdfService.createLabel(
                    ...
            )

            if (parcel.documentsRef == null) {
                logger.error { "No documentsRef found on Parcel." }
                return@run
            }

            val document = fftService.uploadLabel(packJobCreatedEvent.payload.documentsRef,labelBase64)
            if (document == null) {
                logger.error { "No label could be created." }
                return@run
            }

            fftService.setParcelToDone(parcel, createdDelivery, document.id)
        } else {
            logger.error { "Missing parameters in callback. Aborting." }
            return@run
        }
    }
    /**
     * Called from Uber for delivery tracking updates
     */
    @PostMapping("/deliveryStatusCallback")
    suspend fun deliveryStatusCallback(@RequestBody deliveryStatusEvent: DeliveryStatusEvent) = run {
        if (deliveryStatusEvent.status != "pickup_complete") {
            logger.info { "Discarding Uber Event: ${deliveryStatusEvent.status}. Processing stopped." }
            return@run
        }
        logger.info { "Trigger Event ${deliveryStatusEvent.status} received from Uber Direct: ${Gson().toJson(deliveryStatusEvent)}" }

        val ids: List<String> = deliveryStatusEvent.data.external_id?.split("####") ?: emptyList()
        if (ids.size != 2) {
            logger.error { "Wrong count (${ids.size}) of params in Uber external_id." }
            return@run
        }

        val shipmentId: String = ids[0]
        val parcelId: String = ids[1]
        val handoverJob = fftService.getHandoverForParcelInShipment(parcelId,shipmentId)
        if(handoverJob == null){
            throw Exception("No HandoverJob found for Parcel $parcelId after 5 retries!")
        }
        fftService.setHandoverToHandedOver(handoverJob)

        logger.info { "Process completed SUCCESSFULLY for Delivery ${deliveryStatusEvent.id}. Done." }
    }

Deploy service to Google CloudRun

By issuing the command

$> gcloud run deploy headlesscarrierservice

in the console we trigger the deployment of the described cloud run service from our source code. A little time later the service has been build & the container is available under the given URL - in our case: https://headlesscarrierservice-558061097749.europe-west1.run.app

We need to note this host since it is important for the two remaining configuration steps at fulfillmenttools and Uber (see below).

Connect fulfillmenttools to our service creating a Subscription

In order to connect our service to our fulfillmenttools instance we need to create a HTTP subscription for the desired event. This is done according to the documentation by issuing the following call:

curl --location 'https://your.api.fulfillmenttools.com//api/subscriptions/' \
      --header 'Content-Type: application/json' \
      --header 'Authorization: ••••••' \
      --data '{
          "callbackUrl": "https://headlesscarrierservice-558061097748.europe-west1.run.app/requestPickupCallback",
          "event": "PACK_JOB_CREATED",
          "headers": [
              {
                  "key": "Content-Type",
                  "value": "application/json"
              }
          ],
          "name": "Uber Carrier Connector Service - Request Pickup after Picking Completed"
      }'
< 201 Created
{
    "callbackUrl": "https://headlesscarrierservice-558061097748.europe-west1.run.app/requestPickupCallback",
    "event": "PACK_JOB_CREATED",
    "headers": [
        {
            "key": "Content-Type",
            "value": "application/json"
        }
    ],
    "name": "Uber Carrier Connector Service - Request Pickup after Picking Completed",
    "lastModified": "2024-09-24T09:27:53.833Z",
    "id": "b0eee1d0-ae92-4723-8a67-e73ef60d7ac7",
    "created": "2024-09-24T09:27:53.833Z"
}

From now on every PACK_JOB_CREATED event within fulfillmenttools, is being received & processed by our created service.

Configuration of Uber

Uber offers a helpful webhook mechanism similar to the subscription pattern in fulfillmenttools. We can establish a connection from Uber towards the created service by creating a webhook for event.delivery_status events like so in their UI (in german language):

Make sure that you use the correct service URL and endpoint. Once saved Uber will send tracking events to the given endpoint where it is handled by our service.

That's it!

When you now process am order in said facility our mechanism kicks in and performs the creation of the delivery, uploads a self generated PDF to fulfillmenttools & processes the Track & Trace Events.