Troubleshooting
Also see the Vespa FAQ and Vespa support for more help resources.
Vespa.ai and pyvespa
Both Vespa and pyvespa APIs change regularly -
make sure to use the latest version of vespaengine/vespa
by running docker pull vespaengine/vespa and install pyvespa.
python3 -m pip show pyvespa shows current version.
Docker Memory
pyvespa will start a Docker container with 4G memory by default -
make sure Docker settings have at least this.
Use the Docker Desktop settings or docker info | grep "Total Memory" to validate.
Port conflicts / Docker
Some of the notebooks run a Docker container.
Make sure to stop running Docker containers before (re)running pyvespa notebooks -
run docker ps and docker ps -a -q -f status=exited to list containers.
Deployment
Vespa has safeguards for incompatible deployments,
and will warn with validation-override or INVALID_APPLICATION_PACKAGE in the deploy output.
See validation-overrides.
Most often is this due to pyvespa reusing a Docker container instance,
and the fix is to list - docker ps - and remove - docker rm -f <container id> -
the existing Docker containers.
Alternatively, using the Docker Dashboard application.
Then deploy again.
After a deployment, validate status:
- Config server state: http://localhost:19071/state/v1/health
- Container state: http://localhost:8080/state/v1/health
Look for "status" : { "code" : "up"} - both URLs must work before feeding or querying.
Full disk
Make sure to allocate enough disk space for Docker in Docker settings - if writes/queries fail/no results, look in the vespa.log (output in the Docker dashboard):
WARNING searchnode
proton.proton.server.disk_mem_usage_filter Write operations are now blocked:
'diskLimitReached: { action: "add more content nodes",
reason: "disk used (0.939172) > disk limit (0.9)",
stats: { capacity: 50406772736, used: 47340617728, diskUsed: 0.939172, diskLimit: 0.9}}'
Future pyvespa versions might throw an exception in these cases. See Feed block - Vespa stops writes before the disk goes full. Add more disk / clean up, or follow the example to reconfigure for higher usage.
Check number of indexed documents
For query errors, check the number of documents indexed before debugging further:
app.query(body={'yql': 'select * from sources * where true'}).number_documents_indexed.
If this is zero, check that the deployment of the application worked, and the subsequent feeding step.
Too many open files during batch feeding
This is an OS-related issue. There are two options to solve the problem:
- Reduce the number of async connections via the connections parameter:
app.feed_batch(..., connections, ...). - Increase the open file limit:
ulimit -n 10000. Check if the limit was increased withulimit -Sn.