Issue using Gemini Models

[zacsims]

Hello,

I am trying to use the Gemini models with the latest version of paperqa like so:

import os
from paperqa import Settings, ask
from paperqa.settings import AgentSettings

GEMINI_API_KEY = os.getenv('GEMINI_API_KEY')
llm = 'gemini/gemini-1.5-flash'

answer_response = ask(
    "What is Paper-QA?",
    settings=Settings(
        agent=AgentSettings(agent_llm=llm),
        llm=llm,
        summary_llm=llm,
        embedding="gemini/text-embedding-004"
    )
)

However, I am getting this error in response:

BadRequestError: litellm.BadRequestError: VertexAIException BadRequestError - {                                                                                                                                          
             "error": {                                                                                                                                                                                                             
               "code": 400,                                                                                                                                                                                                         
               "message": "* GenerateContentRequest.tools[0].function_declarations[0].parameters.properties: should be non-empty for OBJECT type\n* GenerateContentRequest.tools[0].function_declarations[1].parameters.properties: 
           should be non-empty for OBJECT type\n",                                                                                                                                                                                  
               "status": "INVALID_ARGUMENT"                                                                                                                                                                                         
             }                                                                                                                                                                                                                      
           } 

Attached is the entire output log.

error.txt

[derspotter]

I had a similar problem, but only when using Gemini models as agent llm. As llm for answering they were fine. Right now I am using gemini-2.0-flash-thinking-exp as answer llm and it works perfectly. Just use claude, deepseek or openai as agent llm and you should be fine.
this is what i am using right now:
pqa --settings high_quality
--llm gemini-2.0-flash-thinking-exp
--llm_config '{"model_list":[{"model_name":"gemini-2.0-flash-thinking-exp","litellm_params":{"model":"gemini/gemini-2.0-flash-thinking-exp"}}],"rate_limit":{"gemini-2.0-flash-thinking-exp":"4000000 per 1 minute"}}'
--summary_llm deepseek-chat
--summary_llm_config '{"model_list":[{"model_name":"deepseek-chat","litellm_params":{"model":"deepseek/deepseek-chat"}}]}'
--agent.agent_llm claude-3-5-sonnet-20240620
--agent.agent_llm_config '{"model_list":[{"model_name":"claude-3-5-sonnet-20240620","litellm_params":{"model":"anthropic/claude-3-5-sonnet-20240620"}}],"rate_limit":{"claude-3-5-sonnet-20240620":"40000 per 1 minute"}}'
ask "What is the problem with capital mobility?"

my problem is that it always indexes anew and sometimes that runs into rate limits

[zacsims]

@derspotter That makes sense thanks for the suggestion.

[jamesbraza]

Hi all, this was an interesting issue. I logged a separate issue #796 with the underlying problem.

Just use claude, deepseek or openai as agent llm and you should be fine.

Yes this is the workaround for now, just use another LLM provider that supports empty tool parameters for the agent LLM. The other LLMs (llm, summary_llm) are not impacted by this (since they don't make tool selections).

Thanks for the issue report @zacsims and your help @derspotter.