[firebase_vertexai]: Multi tool generative AI calls fail with server error "Request contains an invalid argument"

[MrCsabaToth]

Is there an existing issue for this?

• I have searched the existing issues.

Which plugins are affected?

Other

Which platforms are affected?

Android

Description

My Generative AI application contains several functions grouped into multiple tools. Whenever when I try to perform a call on the Gemini 1.5 or 2.0 model (regardless of Flash or Pro) equipped with those tools the call fails with server error "Request contains an invalid argument". When I refactor the Tool list to "flatten" all the functions into one unified mixed tool, the call succeeds.

How it should work:

Tool A:
  FunctionDeclaration A1
  FunctionDeclaration A2
Tool B (completely unrelated to A):
  FunctionDeclaration B1
  FunctionDeclaration B2
...

The only way it works now:

Tool (mixed salad of unrelated functions):
  FunctionDeclaration A1
  FunctionDeclaration A2
  FunctionDeclaration B1
  FunctionDeclaration B2
...

I supply the tools during the model isntantiation:

    final model = FirebaseVertexAI.instance.generativeModel(
      model: 'gemini-1.5-flash',
      systemInstruction: Content.text(systemInstruction),
      [getFLattenedFunctionDeclarations(preferences)],
    );
    final chatSession = model.startChat();
    await chatSession.sendMessage(message)

Reproducing the issue

1. Take a simple use case when you have at last two tools each with at least one function.
2. Instantiate a model with these tools.
3. Instantiate a chat session.
4. Send a user message via the chat session.
   The call only survives the call without the server side error if the tools' functions are flattened into a list.

Firebase Core version

3.8.1

Flutter Version

3.27.0

Relevant Log Output

No response

Flutter dependencies

Expand Flutter dependencies snippet

Dart SDK 3.6.0
Flutter SDK 3.27.0
inspector_gadget 1.0.2+1

dependencies:
- adaptive_dialog 2.3.0 [animations collection dynamic_color flutter intersperse macos_ui meta]
- assorted_layout_widgets 10.0.4 [clock flutter matrix4_transform]
- camera 0.11.0+2 [camera_android_camerax camera_avfoundation camera_platform_interface camera_web flutter flutter_plugin_android_lifecycle]
- cloud_functions 5.2.0 [cloud_functions_platform_interface cloud_functions_web firebase_core firebase_core_platform_interface flutter]
- collection 1.19.1
- country_flags 3.1.0 [collection flutter jovial_svg]
- dart_helper_utils 3.3.0 [intl mime meta]
- daylight 3.1.0 [equatable]
- file_picker 8.1.6 [flutter flutter_web_plugins flutter_plugin_android_lifecycle plugin_platform_interface ffi path win32 cross_file web]
- firebase_app_check 0.3.1+7 [firebase_app_check_platform_interface firebase_app_check_web firebase_core firebase_core_platform_interface flutter]
- firebase_auth 5.3.4 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta]
- firebase_core 3.8.1 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_storage 12.3.7 [firebase_core firebase_core_platform_interface firebase_storage_platform_interface firebase_storage_web flutter]
- firebase_ui_storage 2.1.0 [flutter firebase_storage firebase_ui_localizations firebase_ui_shared path uuid file_selector flutter_blurhash]
- firebase_vertexai 1.0.4 [firebase_app_check firebase_auth firebase_core firebase_core_platform_interface flutter http]
- fl_location 5.0.0 [flutter fl_location_platform_interface fl_location_web]
- flex_color_scheme 8.0.2 [flex_seed_scheme flutter meta]
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine]
- flutter_easy_animations 0.0.2 [flutter]
- flutter_layout_grid 2.0.7 [flutter collection quiver]
- flutter_localizations 0.0.0 [flutter intl characters clock collection material_color_utilities meta path vector_math]
- flutter_localized_locales 2.0.5 [flutter intl]
- flutter_tts 4.2.0 [flutter flutter_web_plugins]
- gap 3.0.1 [flutter]
- get_it 8.0.3 [async collection meta]
- heart_rate_flutter 0.0.2 [flutter plugin_platform_interface]
- http 1.2.2 [async http_parser meta web]
- intl 0.19.0 [clock meta path]
- listview_utils_plus 0.0.3 [flutter http]
- media_kit 1.1.11 [meta path image uri_parser collection synchronized uuid http safe_local_storage universal_platform js]
- media_kit_libs_audio 1.0.5 [media_kit_native_event_loop media_kit_libs_android_audio media_kit_libs_ios_audio media_kit_libs_macos_audio media_kit_libs_windows_audio media_kit_libs_linux]
- mime 2.0.0
- number_selector 1.0.7 [flutter]
- objectbox 4.0.3 [collection flat_buffers ffi meta path]
- objectbox_flutter_libs 4.0.3 [flutter objectbox path_provider]
- path 1.9.1
- path_provider 2.1.5 [flutter path_provider_android path_provider_foundation path_provider_linux path_provider_platform_interface path_provider_windows]
- pref 2.8.0 [flutter logging shared_preferences]
- record 5.2.0 [flutter uuid record_platform_interface record_web record_windows record_linux record_android record_darwin]
- smooth_page_indicator 1.2.0+3 [flutter]
- speech_to_text 7.0.0 [flutter speech_to_text_platform_interface json_annotation clock pedantic flutter_web_plugins meta web]
- statemachine 3.3.1 [meta]
- strings 3.1.2 [characters unicode]
- translator 1.0.3+1 [http]
- tuple 2.0.2
- video_player 2.9.2 [flutter html video_player_android video_player_avfoundation video_player_platform_interface video_player_web]
- watch_it 1.6.1 [get_it flutter functional_listener]

dev dependencies:
- build_runner 2.4.13 [analyzer args async build build_config build_daemon build_resolvers build_runner_core code_builder collection crypto dart_style frontend_server_client glob graphs http_multi_server io js logging meta mime package_config path pool pub_semver pubspec_parse shelf shelf_web_socket stack_trace stream_transform timing watcher web_socket_channel yaml]
- flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math leak_tracker_flutter_testing async boolean_selector characters collection leak_tracker leak_tracker_testing material_color_utilities meta source_span stream_channel string_scanner term_glyph vm_service]
- mocktail 1.0.4 [collection matcher test_api]
- objectbox_generator 4.0.3 [objectbox analyzer build collection dart_style glob path source_gen pubspec_parse yaml http pointycastle]
- very_good_analysis 6.0.0

dependency overrides:
- collection 1.19.1
- mime 2.0.0
- path 1.9.1

transitive dependencies:
- _fe_analyzer_shared 76.0.0 [meta]
- _flutterfire_internals 1.3.47 [collection firebase_core firebase_core_platform_interface flutter meta]
- _macros 0.3.3
- analyzer 6.11.0 [_fe_analyzer_shared collection convert crypto glob macros meta package_config path pub_semver source_span watcher yaml]
- animations 2.0.11 [flutter]
- appkit_ui_element_colors 1.0.0 [equatable flutter plugin_platform_interface]
- archive 4.0.1 [crypto path posix]
- args 2.6.0
- async 2.11.0 [collection meta]
- boolean_selector 2.1.1 [source_span string_scanner]
- build 2.4.1 [analyzer async convert crypto glob logging meta package_config path]
- build_config 1.1.1 [checked_yaml json_annotation path pubspec_parse yaml]
- build_daemon 4.0.2 [built_collection built_value crypto http_multi_server logging path pool shelf shelf_web_socket stream_transform watcher web_socket_channel]
- build_resolvers 2.4.2 [analyzer async build collection convert crypto graphs logging package_config path pool pub_semver stream_transform yaml]
- build_runner_core 7.3.2 [async build build_config build_resolvers collection convert crypto glob graphs json_annotation logging meta package_config path pool timing watcher yaml]
- built_collection 5.1.1
- built_value 8.9.2 [built_collection collection fixnum meta]
- camera_android_camerax 0.6.10 [async camera_platform_interface flutter meta stream_transform]
- camera_avfoundation 0.9.17+5 [camera_platform_interface flutter stream_transform]
- camera_platform_interface 2.8.0 [cross_file flutter plugin_platform_interface stream_transform]
- camera_web 0.3.5 [camera_platform_interface flutter flutter_web_plugins stream_transform web]
- characters 1.3.0
- checked_yaml 2.0.3 [json_annotation source_span yaml]
- clock 1.1.1
- cloud_functions_platform_interface 5.5.40 [firebase_core flutter meta plugin_platform_interface]
- cloud_functions_web 4.10.5 [cloud_functions_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins]
- code_builder 4.10.1 [built_collection built_value collection matcher meta]
- convert 3.1.2 [typed_data]
- cross_file 0.3.4+2 [meta web]
- crypto 3.0.6 [typed_data]
- csslib 1.0.2 [source_span]
- dart_style 2.3.7 [analyzer args collection package_config path pub_semver source_span]
- dynamic_color 1.7.0 [flutter flutter_test material_color_utilities]
- equatable 2.0.7 [collection meta]
- fake_async 1.3.1 [clock collection]
- ffi 2.1.3
- file 7.0.1 [meta path]
- file_selector 1.0.3 [file_selector_android file_selector_ios file_selector_linux file_selector_macos file_selector_platform_interface file_selector_web file_selector_windows flutter]
- file_selector_android 0.5.1+12 [file_selector_platform_interface flutter plugin_platform_interface]
- file_selector_ios 0.5.3+1 [file_selector_platform_interface flutter]
- file_selector_linux 0.9.3+2 [cross_file file_selector_platform_interface flutter]
- file_selector_macos 0.9.4+2 [cross_file file_selector_platform_interface flutter]
- file_selector_platform_interface 2.6.2 [cross_file flutter http plugin_platform_interface]
- file_selector_web 0.9.4+2 [file_selector_platform_interface flutter flutter_web_plugins web]
- file_selector_windows 0.9.3+3 [cross_file file_selector_platform_interface flutter]
- firebase_app_check_platform_interface 0.1.0+41 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface]
- firebase_app_check_web 0.2.0+3 [_flutterfire_internals firebase_app_check_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins web]
- firebase_auth_platform_interface 7.4.10 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_auth_web 5.13.5 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser meta web]
- firebase_core_platform_interface 5.3.1 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.18.2 [firebase_core_platform_interface flutter flutter_web_plugins meta web]
- firebase_storage_platform_interface 5.1.34 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_storage_web 3.10.6 [_flutterfire_internals async firebase_core firebase_core_web firebase_storage_platform_interface flutter flutter_web_plugins http meta web]
- firebase_ui_localizations 1.13.0 [flutter flutter_localizations path]
- firebase_ui_shared 1.4.1 [flutter]
- fixnum 1.1.1
- fl_location_platform_interface 5.1.0 [flutter meta vector_math plugin_platform_interface]
- fl_location_web 5.0.0 [flutter flutter_web_plugins fl_location_platform_interface web]
- flat_buffers 23.5.26
- flex_seed_scheme 3.4.1 [collection flutter meta]
- flutter_blurhash 0.8.2 [flutter]
- flutter_plugin_android_lifecycle 2.0.23 [flutter]
- flutter_web_plugins 0.0.0 [flutter characters collection material_color_utilities meta vector_math]
- frontend_server_client 4.0.0 [async path]
- functional_listener 4.1.0 [flutter]
- glob 2.1.2 [async collection file path string_scanner]
- gradient_borders 1.0.1 [flutter]
- graphs 2.3.2 [collection]
- html 0.15.5 [csslib source_span]
- http_multi_server 3.2.1 [async]
- http_parser 4.1.1 [collection source_span string_scanner typed_data]
- image 4.4.0 [archive meta xml]
- intersperse 2.0.0
- io 1.0.5 [meta path string_scanner]
- jovial_misc 0.9.0 [async collection convert meta]
- jovial_svg 1.1.24 [xml collection args vector_math jovial_misc http meta flutter]
- js 0.6.7 [meta]
- json_annotation 4.9.0 [meta]
- leak_tracker 10.0.7 [clock collection meta path vm_service]
- leak_tracker_flutter_testing 3.0.8 [flutter leak_tracker leak_tracker_testing matcher meta]
- leak_tracker_testing 3.0.1 [leak_tracker matcher meta]
- lists 1.0.1 [meta]
- logging 1.3.0
- macos_ui 2.0.9 [flutter macos_window_utils gradient_borders appkit_ui_element_colors equatable]
- macos_window_utils 1.6.1 [flutter]
- macros 0.1.3-main.0 [_macros]
- matcher 0.12.16+1 [async meta stack_trace term_glyph test_api]
- material_color_utilities 0.11.1 [collection]
- matrix4_transform 4.0.1 [flutter]
- media_kit_libs_android_audio 1.3.6 [flutter plugin_platform_interface]
- media_kit_libs_ios_audio 1.1.4 [flutter plugin_platform_interface]
- media_kit_libs_linux 1.1.3 [flutter]
- media_kit_libs_macos_audio 1.1.4 [flutter plugin_platform_interface]
- media_kit_libs_windows_audio 1.0.9 [flutter]
- media_kit_native_event_loop 1.0.9 [flutter]
- meta 1.15.0
- package_config 2.1.1 [path]
- path_provider_android 2.2.15 [flutter path_provider_platform_interface]
- path_provider_foundation 2.4.1 [flutter path_provider_platform_interface]
- path_provider_linux 2.2.1 [ffi flutter path path_provider_platform_interface xdg_directories]
- path_provider_platform_interface 2.1.2 [flutter platform plugin_platform_interface]
- path_provider_windows 2.3.0 [ffi flutter path path_provider_platform_interface]
- pedantic 1.11.1
- petitparser 6.0.2 [meta]
- platform 3.1.6
- plugin_platform_interface 2.1.8 [meta]
- pointycastle 3.9.1 [collection convert js]
- pool 1.5.1 [async stack_trace]
- posix 6.0.1 [ffi meta path]
- pub_semver 2.1.5 [collection meta]
- pubspec_parse 1.3.0 [checked_yaml collection json_annotation pub_semver yaml]
- quiver 3.2.2 [matcher]
- record_android 1.3.0 [flutter record_platform_interface]
- record_darwin 1.2.2 [flutter record_platform_interface]
- record_linux 0.7.2 [flutter record_platform_interface]
- record_platform_interface 1.2.0 [flutter plugin_platform_interface]
- record_web 1.1.4 [flutter flutter_web_plugins web record_platform_interface]
- record_windows 1.0.4 [flutter record_platform_interface]
- safe_local_storage 1.0.2 [path synchronized]
- shared_preferences 2.3.3 [flutter shared_preferences_android shared_preferences_foundation shared_preferences_linux shared_preferences_platform_interface shared_preferences_web shared_preferences_windows]
- shared_preferences_android 2.3.4 [flutter shared_preferences_platform_interface]
- shared_preferences_foundation 2.5.3 [flutter shared_preferences_platform_interface]
- shared_preferences_linux 2.4.1 [file flutter path path_provider_linux path_provider_platform_interface shared_preferences_platform_interface]
- shared_preferences_platform_interface 2.4.1 [flutter plugin_platform_interface]
- shared_preferences_web 2.4.2 [flutter flutter_web_plugins shared_preferences_platform_interface web]
- shared_preferences_windows 2.4.1 [file flutter path path_provider_platform_interface path_provider_windows shared_preferences_platform_interface]
- shelf 1.4.2 [async collection http_parser path stack_trace stream_channel]
- shelf_web_socket 2.0.1 [shelf stream_channel web_socket_channel]
- sky_engine 0.0.0
- source_gen 1.5.0 [analyzer async build dart_style glob path source_span yaml]
- source_span 1.10.0 [collection path term_glyph]
- speech_to_text_platform_interface 2.3.0 [flutter meta plugin_platform_interface]
- sprintf 7.0.0
- stack_trace 1.12.0 [path]
- stream_channel 2.1.2 [async]
- stream_transform 2.1.0
- string_scanner 1.3.0 [source_span]
- synchronized 3.3.0+3
- term_glyph 1.2.1
- test_api 0.7.3 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph]
- timing 1.0.2 [json_annotation]
- typed_data 1.4.0 [collection]
- unicode 0.3.1 [lists]
- universal_platform 1.1.0
- uri_parser 2.0.2 [path safe_local_storage]
- uuid 4.5.1 [crypto sprintf meta fixnum]
- vector_math 2.1.4
- video_player_android 2.7.16 [flutter video_player_platform_interface]
- video_player_avfoundation 2.6.5 [flutter video_player_platform_interface]
- video_player_platform_interface 6.2.3 [flutter plugin_platform_interface]
- video_player_web 2.3.3 [flutter flutter_web_plugins video_player_platform_interface web]
- vm_service 14.3.0
- watcher 1.1.0 [async path]
- web 1.1.0
- web_socket 0.1.6 [web]
- web_socket_channel 3.0.1 [async crypto stream_channel web web_socket]
- win32 5.9.0 [ffi]
- xdg_directories 1.1.0 [meta path]
- xml 6.5.0 [collection meta petitparser]
- yaml 3.1.2 [collection source_span string_scanner]

Additional context and comments

No response

[Lyokone]

Hello @MrCsabaToth thanks for the detailed report. Could you provide a complete sample so we can reproduce your issue on our end?

[google-oss-bot]

Hey @MrCsabaToth. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[MrCsabaToth]

Sorry I'm bogged down with work. I'll try to carve out some time during the weekend or next week.

[MrCsabaToth]

I need a little more time please

[MrCsabaToth]

https://github.com/CsabaConsulting/issue_16883

[MrCsabaToth]

https://github.com/CsabaConsulting/issue_16883/blob/2b6ffe74863c35f515e2899bf33ee0144f3c30df/lib/ai_service.dart#L15

[MichaelVerdon]

Hey there, many thanks for the reproduction repo however I have been unable to run it but I have been able to test it on the example application. It is known on Gemini model 1 and 1.5, there are issues with putting in multiple tools as parameters: GoogleCloudPlatform/generative-ai#1561 So out of Gemini 1.5 and 2 make sure you use Gemini 2 as older models do not support multi tooling. However, after trying this out myself on the example app making some new tools, it does not appear that we support multi tooling yet in the FlutterFire implementation, so I will keep this ticket open and I can only suggest using your multiple tools as one as a workaround for now.