Cloud not read served logs

[DevDan99]

Official Helm Chart version

1.11.0 (latest released)

Apache Airflow version

2.7.0

Kubernetes Version

Not use

Helm Chart configuration

Default setting

Docker Image customizations

No customization

What happened

When running a DAG, the process throws an error within a few seconds.
Error message: Cloud-served logs not read: Request URL is missing an 'http://' or 'https://' protocol.
I'm not able to identify the problem so I think it's a bug, as a few days ago the Dag was running normally.
I've already tried reinstalling the airflow image, installing old versions, but the error persists.
The server that hosts airflow does not have any type of proxy or firewall blocking

Error image:

What you think should happen instead

The correct execution, which was carried out a few days ago, is after the DAG starts, it makes a request on a website, captures the information and uploads it to my data lake.

How to reproduce

In my case, as soon as I run any DAG, which makes a request on a website, the error happens.

Anything else

Graphic Image:

My code:

from airflow import DAG #lib da Dag no airflow
from datetime import datetime #lib para obter data e hora
from airflow.operators.python import PythonOperator #Lib para cria dag
from minio import Minio #lib para interagir com o Min IO
import pandas as pd #tratamento de dados
import requests #faz requisições HTML
import io # criar o arqui temporario na memoria
import json


def Capta_dolar (): #faz requisição HTTP e obtem o JSON
    url = "https://olinda.bcb.gov.br/olinda/servico/PTAX/versao/v1/odata/CotacaoMoedaPeriodo(moeda=@moeda,dataInicial=@dataInicial,dataFinalCotacao=@dataFinalCotacao)?@moeda='USD'&@dataInicial='01-01-2022'&@dataFinalCotacao='12-31-2090'&$top=10000&$format=json&$select=cotacaoCompra,cotacaoVenda,dataHoraCotacao,tipoBoletim"
    response = requests.get(url)
    dados= json.loads(response.content) #abrindo o arquivo que foi pego
    key_select= ['cotacaoCompra','cotacaoVenda','dataHoraCotacao','tipoBoletim'] #cria lista de chaves escolhidas e organizar os dados dos mesmos tipos
    filtro_dados= [{key: D_orig[key] for key in key_select} for D_orig in dados['value']] #para cada dicionario no Json, é criado um novo dicionario, contendo apenas as chaves filtradas
    df= pd.DataFrame(filtro_dados)
    return df

def Transforma_CSV(ti): # Transforma o JSON em CSV
    df= ti.xcom_pull(task_ids='Capta_dolar')
    csv= df.to_csv(index=False)
    return csv

def Upload_Minio(ti):
    csv = ti.xcom_pull(task_ids= 'Transforma_CSV') #pega dados da task aterior
    
    config = {
        "dest_bucket": "processed",
        "minio_endpoint": "10.2.0.28:9000",
        "minio_user": "AirFl",
        "minio_password": "dBCvoOhizLpyvFZFWXA7ceqIo4J7XJYRhyD9y"
    } #cria um dicionairo para salvar as informações de conexao
    
    minio_client = Minio(
        config["minio_endpoint"],
        access_key=config["minio_user"],
        secret_key=config["minio_password"],
        secure=False
    ) # faz a conexão com Minio
    
    bucket_name= 'dadospublicos'
    file_name= 'Cot_dolar.csv'
    csv_bytes= csv.encode('utf-8') #converte str em bytes

    minio_client.put_object(
        bucket_name,
        file_name,
        io.BytesIO(csv_bytes), #passa o Byte em um objeto BytesIO
        len(csv_bytes),
        content_type= 'text/csv'
    ) # realiza o Upload para o Bucket no MinIo

with DAG('Capta_Dolar',start_date= datetime(2023,9,27), schedule_interval='@daily', catchup= False) as dag:

    Capta_dolar= PythonOperator(
        task_id= 'Capta_dolar',
        python_callable= Capta_dolar 
    )

    Transforma_CSV= PythonOperator(
        task_id= 'Transforma_CSV',
        python_callable= Transforma_CSV
    )

    Upload_Minio= PythonOperator(
        task_id= 'Upload_Minio',
        python_callable= Upload_Minio

    )

    Capta_dolar >> Transforma_CSV >> Upload_Minio

I've already opened a discussion about the problem, but no one responds to me.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jscheffl]

This sounds more like a deployment issue in core, not related to helm-chart. I assume to judge this problem report more details are needed.

If a scheduled task runs into error w/o logs being served, two cases can be the root of the problem, typically related to the deployment if broken in general - best is to double check with one of the example DAGs. Can you please check:

1. Do you see errors in the console of the scheduler or worker indicating a general issue (like DB not accessible, out of memory etc.)?
2. Can you check your log back-end if really not logs are being created or if this is a display problem? If logs are existing in your (however deployed) storage back-end, they might reveal more details.

[DevDan99]

1. When I run the test DAGs or example DAGs, they run successfully. So I don't have a memory problem. I made some DAGs and they also ran successfully, but, when I run the dag to upload to MinIO, it presents this error.

Would you have any suggestions on where I can configure my log to communicate with Minio?

2. Yes, I agree with you, if my logs run within Airflow, in the example DAGs, perhaps the error is not in Airflow. Apparently the error only appears when I run DAGs that upload to MinIO. The log error is the one I showed in the print, no other error appears. Do you know if I can look at the MinIO input log?

Is there any other information I can provide that may help me?
Thanks for answering!

[DevDan99]

Is there a way to configure logging in Python code within the DAG?

[jscheffl]

Logging in the DAG follows the Python standard from the loggingpackage. You ned to consider that "global" code is not executed when you run inside the task (body) - in this case you need to change Python logging options inside the tasks.

But I did not understand the problem, usually no adjustments for the logging are needed.

[DevDan99]

So my problem is this. I made a Dag, the same when it is run in Airflow, it runs for about 2 seconds and gives an error, when I go to the Airflow log tab, the following message appears: *** Could not read served logs: Request URL is missing an ' http://' or 'https://' protocol.
This Dag makes a request to a URL, takes the information, transforms it into CSV and sends it to a MinIO Bucket (My data lake), I use an AWS S3 connection. As the Log doesn't give me much information I don't know what's going wrong. To make matters worse, this process was working successfully a few days ago, and suddenly stopped working.

That's why I think there's a problem with the Log.
Today I use all the default settings in the .yml file (I downloaded the file from the airflow website), I did the entire installation using Docker.
Can you help me?

[jscheffl]

No, I assume from far far away it is hard to help. To make better support you need to really understand your setup and details. In such case I assume I would seek for consultancy in the community or look for a company that supports you in such cases.
This here is all OpenSource, powered by volunteers. You could check the #jobs channel in Slack or look for commercial supporters like Astronomer or similar.

[potiuk]

Indeed it looks like troubleshooting, converted to discussion.

[rishabh208gupta]

Is there any update on this? Even I am facing the same issue

[soravispr]

Any solution to this yet? I'm having this exact issue. Running Airflow on a K8s cluster.

[Gunnnn]

Have same problem on docker.

[potiuk]

@rishabh208gupta @Gunnnn @soravispr - if you add to the discussion and explain your case and provide more evidences, then there is a chance, yes, that someone will be able to analyse your case and circumstances and maybe even come with a solution or workaround or hypothesis why it might happen.

But without it, it's unlikely.

[Gunnnn]

Solved. My case was simple: airflow-worker docker service has lost its volumes during code change and couldn't write logs.
Nethertheless error telling smth about 'http' is far away from my real error.

[potiuk]

Well. I think that's a great opportunity so that you can reproduce this situation and make a PR that will handle it better. Generally when your environment (which deployment manager - i.e. you) is responsible to manage, then many assumptions the software makes might be not fullfilled any more and it's often imposible to foresee all the ways the environment (which again airflow software running in makes some assumptions about) can get broken. So yes, sometimes, the unforeseen ways things are broken can manifest in strange errors - that's why the basic requirement of deploying airflow (and responsibility of deployment manager as described in https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html ) is to setup monitoring for the whole system and be able to see all the cases where environment is broken. Which is pretty fair expectation if you take into account that this is free software, that you can install and it comes without paid support and guarantees - where somoene deploying the software get what they pay for, basic software that has some expectations and then they have to make sure the expectations are met.

Speaking of which - if you think that such case is likely and others could benefit from that, that's a great opportunity to give back to the community for the free software, reproduce the issue and make a PR where error logs will be clearer. this way you have a chance to become one of the 2800+ contributors to Airflow - many of them doing exactly this - experiencing a problem, solving it and deciding to contribute back by PR that will help others to avoid similar issue.

And I heartily encourage you to do so. That's much better way to handle such situation than to just complain about misleading error messages. Unlike just complaining, it has a chance of moving a needle and improving things and making a world a better place (which many people here do).

[trlopes1974]

I had this issue and in my case was a missing library
you should check your imports.
somewhere there is one that is not working or not installed.

[swethabw]

Which library?
Could you detail on this?

[ledikari]

I'm very curious too, I encounter this issue using MWAA

[Krishna4802]

I'm facing the same issue when trying to run a dag. it is failing automatically after the 10 minutes of queue.
Error I'm getting is
*** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol

[fabinhojorge]

Failing after 10 minutes of queue probably is related to task_queued_timeout parameter that the default is 600sec.

[christopher-leigh]

I am experiencing the same issue:

*** Unable to read remote logs from Cloudwatch (log_group: airflow-combustion-dev-Task, log_stream: dag_id=dbt_build/run_id=manual__2024-02-21T18_46_49.436031+00_00/task_id=pre_dbt_workflow/attempt=1.log)
*** An error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist.

*** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.

[swethabw]

I am also facing the same issue. Any update from your side?

[youngtai]

I needed to use the CloudFormation template provided here to create a VPC that an MWAA environment could function in: https://docs.aws.amazon.com/mwaa/latest/userguide/vpc-create.html
I think the key thing is that the private subnets must be connected to NAT gateways, which my original VPC did not have. There may be some other properties that are necessary though.

[youngtai]

And I just found docs outlining the VPC requirements: https://docs.aws.amazon.com/mwaa/latest/userguide/networking-about.html#networking-about-overview

[simond]

Recently had to resolve this issue. We are using the kubernetes executor and our pod was running an old version of our airflow image. The Worker pod would fail (because the code was importing a package that wasn't on the worker) and the the scheduler had no idea what was going on. I assume this is because the pod would fail and the logs never got written.

[ddeokgu]

We have same problem on docker
Is this main cause of this problem Python dags code or Airflow itself?

[ImpressionableRaccoon]

@potiuk

We have the same error and this is scheduler logs:

Celery Task ID: TaskInstanceKey(dag_id='hello_world_dag', task_id='hello_world', run_id='scheduled__2024-03-14T11:10:00+00:00', try_number=1, map_index=-1)
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 201, in send_task_to_executor
    result = task_to_run.apply_async(args=[command], queue=queue)
  File "/home/airflow/.local/lib/python3.8/site-packages/celery/app/task.py", line 594, in apply_async
    return app.send_task(
  File "/home/airflow/.local/lib/python3.8/site-packages/celery/app/base.py", line 798, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/home/airflow/.local/lib/python3.8/site-packages/celery/app/amqp.py", line 517, in send_task_message
    ret = producer.publish(
  File "/home/airflow/.local/lib/python3.8/site-packages/kombu/messaging.py", line 186, in publish
    return _publish(
  File "/home/airflow/.local/lib/python3.8/site-packages/kombu/connection.py", line 556, in _ensured
    return fun(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/kombu/messaging.py", line 208, in _publish
    return channel.basic_publish(
  File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 614, in basic_publish
    return self._put(routing_key, message, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/sqlalchemy/__init__.py", line 162, in _put
    obj = self._get_or_create(queue)
  File "/home/airflow/.local/lib/python3.8/site-packages/kombu/transport/sqlalchemy/__init__.py", line 138, in _get_or_create
    obj = self.session.query(self.queue_cls) \
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2824, in first
    return self.limit(1)._iter().first()
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
    result = self.session.execute(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1716, in execute
    conn = self._connection_for_bind(bind)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1555, in _connection_for_bind
    return self._transaction._connection_for_bind(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 724, in _connection_for_bind
    self._assert_active()
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 597, in _assert_active
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: This session is in 'prepared' state; no further SQL can be emitted within this transaction.

Also hostname field is empty in Task Instance Attributes

---

Version: v2.7.1
Git Version: .release:b8c416681c529aad3ef744c193f6e0435c4d0d93

[potiuk]

It usually signifies problem with your DNS. If your DNS is not properly configured or has not enough resources you might not get the right host name. You can fix it by fixing your DNS used internally by your Docker configuration, you can also change hostname_callable configuration as described in https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable

[Pratheesh-kumar]

i fixed it by giving write permission to the logs directory

[ledikari]

issue still randomly occur on Apache Ariflow 2.9.2 Using MWAA servce.