Versions Compared

Old Version 1

changes.mady.by.user Prashant Jaikumar

Saved on

compared with

New Version Current

changes.mady.by.user Robin Rielley

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Error No discoverable found for request POST /v3/namespaces/system/apps/pipeline/services/studio/methods/v1/contexts/default/validations/stage HTTP/1.1 is seen when validating pipelines.

Symptom(s)

Pipeline studio Studio status is red (System Admin > Management).

...

Restart the pipeline studio service

  • Navigate to System Admin - > Configuration - > Make HTTP calls.

  • Make a POST call to namespaces/system/apps/pipeline/services/studio/start.

  •  Monitor Monitor the Pipeline Studio status in the System Admin page. You should be able to validate your pipeline when it turns green.

...

Log lines like the following in App Fabric logs indicate that Pipeline Studio service failed to start:

Code Block
2020-06-17 12:18:10,075 - DEBUG [program.status:i.c.c.k.r.KubeTwillRunnerService@223] - Monitoring application service.system.pipeline.studio with run 4627ebab-8437-4cf1-ac4e-2b6496aae72c starts in 300 SECONDS
[...]
2020-06-17 12:23:10,147 - INFO  [OkHttp https://10.114.64.33/api/v1/namespaces/default/configmaps/cdap-dfusion-sl-edwea-master-dev-service-system-pipeline-studio-4627ebab-8437-4cf1-ac4e-2b6496aae72c:i.c.c.i.a.r.d.AbstractTwillProgramController@77] - Twill program terminated: program_run:system.pipeline.-SNAPSHOT.service.studio.9832b5d7-b094-11ea-b4ba-da2f27192df4, twill runId: 4627ebab-8437-4cf1-ac4e-2b6496aae72c

...

  • any pods are not in RUNNING state.

  • App Fabric pod is running and hasn’t restarted several times.

  • Pipeline studio Studio pod is running.

Pipeline

...

Studio pod is not running

If pipeline studio the Pipeline Studio pod is in PENDING state, kubectl descibe pod/<pipeline-studio-pod-id> will tell you why it’s not running. The most likely cause is pod scheduling issues, indicated by errors like the following:

...

When the command returns, check kubectl get pods to make sure that all pods eventually transition to RUNNING state.

Orphaned pod

If the Pipeline Studio transitions get pod gets stuck in PENDING state for 5 minutes and then gets deleted, that means the cluster doesn’t have enough resources to schedule the pod.

...

In this example, there are 2 Dataprep pods:

Code Block
cdap-dfusion-sl-edwea-master-dev-service-system-dataprep-s2jv22   1/1     Running   0          25m
cdap-dfusion-sl-edwea-master-dev-service-system-dataprep-sb94f4   1/1     Running   0          24m

Remediation:

  • Stop the Dataprep service.

    • Get the router service name from kubectl get services

    • Stop the Dataprep service using kubectl exec -it <app-fabric-pod-name> --curl -X POST http://<router-service-name>:11015/v3/namespaces/system/apps/dataprep/services/service/stop

  • Delete Dataprep deployment to remove the orphaned Dataprep pod.

    • Get Dataprep deployment name from kubectl get deployment

    • Delete the deployment using kubectl delete deployment/<dataprep-deployment-name>

  • Verify that Dataprep pods don’t show up in kubectl get pods.

  • Start the Dataprep service: kubectl exec -it <app-fabric-pod-name> -- curl -X POST http://<router-service-name>:11015/v3/namespaces/system/apps/dataprep/services/service/start

  • Start the Pipeline Studio service: kubectl exec -it <app-fabric-pod-name> -- curl -X POST http://<router-service-name>:11015/v3/namespaces/system/apps/pipeline/services/studio/start

  • Verify that all pods and services are up. Check kubectl get pods, kubectl get services.

...