Debugging Tooling¶
Grapl has several tools to aid in debugging issues.
Grapl Debugger Container
Distributed Tracing
Grapl Debugger Container¶
This is a container that can attach to running containers and includes debug tools inside. See the debugger docs for details.
Distributed Tracing¶
We currently have tracing enabled for local grapl. Our current backend is Jaeger
Usage¶
Run
make up
In your browser go to
http://localhost:16686
. You should see the Jaeger front-end.On the left side, are search options.
Select the service you’re interested in and click search. You can also use any of the additional filters on the left (such as http code, etc). If your service does not appear, it’s possible that a) it doesn’t have any traffic (ie the web ui needs a web request), b) there are no traces from within the Lookback window if
In the center a list of traces will appear.
Click on one. You will go to a page with detailed trace information, including performance data for each span.
Click on a span, and then click on tags to get more detailed information, such as http code, etc
On the top-right, there is a drop-down menu with Trace Timeline selected. Clicking on it will provide a few additional options
Tracing pulumi¶
To have pulumi send traces to Jaeger run
WITH_PULUMI_TRACING=1 make $MAKECOMMND
where $MAKECOMMAND can be up
or
test-e2e
Tracing docker buildx bake¶
docker buildx supports sending traces to a backend. Since we build prior to
running Jaeger, you will need to explicitly set Jaeger up, either via running
make up
first or by running it manually in either docker or as a standalone
nomad job.
This tracing is meant to help debug docker build issues including performance issues.
Warning!
This generates a LOT of traces, enough to potentially crash Jaeger.
This slows down the build process significantly :(
To run tracing for the docker build:
Do a one-time setup
make setup-docker-tracing
if you haven’t already.Run
WITH_TRACING=1 make build-local-infrastructure
or any other build command that uses bake such asbuild-test-e2e
. Alternatively, you can run traces for individual services viadocker buildx bake --file $FILE --builder=grapl-tracing-builder