Convert remaining providers to the new structure

[potiuk]

We have now a number of providers converted to the new structure. We have almost 90 of them to be converted. And we have automated scripts that do bulk of the work. This was part of #44511

Process of the conversion of each of the provider is easy - and described in https://github.com/apache/airflow/blob/main/dev/moving_providers/README.md

The devlist announcement: https://lists.apache.org/thread/dzbj5yx5kwpbwyr5yscp4wnlsp6p9v8l

The list below keeps track of the conversion. The list is generated via:

find providers/src -name 'provider.yaml' | sed 's/providers\/src\/airflow\/providers/ - [ ] /' | sed s'/\/provider.yaml//'  | sor

• /amazon
• /apache/beam
• /apache/cassandra
• /apache/drill
• /apache/druid
• /apache/flink
• /apache/hdfs
• /apache/hive
• /apache/impala
• /apache/kafka
• /apache/kylin
• /apache/livy
• /apache/pig
• /apache/pinot
• /apache/spark
• /apprise
• /arangodb
• /asana
• /atlassian/jira
• /cloudant
• /cncf/kubernetes
• /cohere
• /common/compat
• /common/io
• /databricks
• /datadog
• /dbt/cloud
• /dingding
• /discord
• /docker
• /elasticsearch
• /exasol
• /fab
• /facebook
• /ftp
• /github
• /google
• /grpc
• /hashicorp
• /http
• /imap
• /influxdb
• /jdbc
• /jenkins
• /microsoft/azure
• /microsoft/mssql
• /microsoft/psrp
• /microsoft/winrm
• /mongo
• /mysql
• /neo4j
• /odbc
• /openai
• /openfaas
• /openlineage
• /opensearch
• /opsgenie
• /oracle
• /pagerduty
• /papermill
• /pgvector
• /pinecone
• /postgres
• /presto
• /qdrant
• /redis
• /salesforce
• /samba
• /segment
• /sendgrid
• /sftp
• /singularity
• /slack
• /smtp
• /snowflake
• /sqlite
• /ssh
• /tableau
• /telegram
• /teradata
• /trino
• /vertica
• /weaviate
• /yandex
• /ydb
• /zendesk

[rawwar]

I'll work on Zendesk, ydb

[Prab-27]

I'm working on Trino, Apprise

[eladkal]

Lets please make sure we merge open PRs for the providers (if we can) before the conversion.

[vatsrahul1001]

PR for Weaviate.

[potiuk]

Lets please make sure we merge open PRs for the providers (if we can) before the conversion.

Yeah before attempting to convert a particular provider It would be great to check if there are outsanding PRs. I am going through the PRs from the last few weeks to see if they apply - but if anyone will want to take on a provider - just checking if there are no related PRs before will do the job :)

[josix]

I'm working on openai and mongodb

[potiuk]

Unfortunately a log of those result in conflicts if they are not done in sequence - so you will have to rebase those that have conflicts :(

[potiuk]

We really need to batch the providers now - seems that many of them can be fully automated - so I suggest to select a range of those providers (you can provide multiple providers as input to the move script) and submit them together. That will make resolving of the conflict a bit easier.

[rawwar]

Working on sqlite, redis, vertica - #46101 - All tests passed

[vatsrahul1001]

We really need to batch the providers now - seems that many of them can be fully automated - so I suggest to select a range of those providers (you can provide multiple providers as input to the move script) and submit them together. That will make resolving of the conflict a bit easier.

Yes I have merged mysql odbc jenkins pagerduty providers in PR

[amoghrajesh]

There should not really be an issue related to the migration but I'd prefer not batching them. In case there is an issue, reverting the batch would be uglier than individually isolating and debugging

[jason810496]

Working on

• /apache/pig Move Apache Pig to new provider structure #46105
• /apache/pinot Move Apache Pinot to new provider structure #46106
• /apache/spark
• /apache/kafka
• /apache/kylin
• /apache/livy

[potiuk]

There should not really be an issue related to the migration but I'd prefer not batching them. In case there is an issue, reverting the batch would be uglier than individually isolating and debugging

Yeah. Resolving the conflicts is not that bad and actually the more providers we migrate, the less likely conflicts will be - becase all the conflicting stuff is only when you have adjacent changes in the file - i.e. two providers migrated separatelyt that are "neighbours".

I think our aim should be to migrate providers semi-randomly :)

[Prab-27]

Working on
/exasol
/influxdb
/apache/impala
/dingding
/apache/cassandra
/yandex