DEV Community

Cover image for 😡 Agonies, Despair and self hosted Redash BI server on Microsoft Azure cloud platform 😋 part 2

😡 Agonies, Despair and self hosted Redash BI server on Microsoft Azure cloud platform 😋 part 2

alxizr on June 21, 2021

Hello everyone, Thank you for joining in today for part 2 to talk about the open source BI tool named Redash and how to set up a self hosted Redas...
Collapse
 
reconnect profile image
Reconnect

Thanks for sharing the post!! really well written. I am having trouble connecting to the postgres on azure. As you mentioned "Pay attention that we do specify the name of the actual database name that we are looking to get access to and its 'redash' on purpose". I don't see redash database on azure postgres. Even changing the REDASH_DATABASE_URL in the local instance doesn't connect to azure postgres for replication. Can you please advise. As Redash is phasing out, we need to self host Redash. Thanks

Collapse
 
alxizr profile image
alxizr • Edited

@formerlyknownas
@reconnect

I must say that i am having trouble understanding the real issue. Is it you guys not being able to connect to the Azure managed Postgresql instance with the connection string or are you not able to connect to the database instance within the Postgress database you created in Azure ?

Collapse
 
alxizr profile image
alxizr • Edited

@formerlyknownas
@reconnect

The next thing I will suggest is to understand firstly where is the 'redash' named db coming from, as i talked about it in the first part of this series, this db is auto generated when you run the setup script. Have a read on both parts and pay attention to the specific stage where it is created.

let me know if you guys figured it out.

Collapse
 
reconnect profile image
Reconnect

Hi, It seemed the IP was blocking, so can connect to azure postres and created redash DB and was able to execute docker-compose -f docker-compose.yml run --rm server create_db. I have made some changes to yml to deploy Redash V10. it runs fine in local setup but doesn't run in Azure? Any advise deploying Redash v10 on azure as Celery has been replaced with RQ in redash V10 from Redash V8. Using image redash/redash:10.0.0-beta.b49597.

Thread Thread
 
alxizr profile image
alxizr

@reconnect

Hi,
I'm glad to see that we found the issue easily and you were able to load your cloud environment setup. Regarding Redash v10, I am not familiar with it and I checked docker hub couple minutes ago and saw it was released 2 weeks ago and it is in preview mode. As a rule of thumb I never use the latest version of anything even with the tech stack I know, love and use every day. The headache is not worth the trouble especially if you have something in production already working. That said, if there is a v10 then there should be a migration guide over on the redash official website. I left a link in the article. Take my word for what it's worth. We've been there, me and my team, it's not easy. Stick to the version that works

Best of luck

Thread Thread
 
reconnect profile image
Reconnect

Hi, Thanks for the advise. I just realized that for Redash Version 8 deployment(followed as described in your article), after following all the steps (and do want to mention that its works fine on local machine) getting the following errors on Azure:
ERROR - Exception in multi-container config parsing: Exception: System.NullReferenceException, Msg: Object reference not set to an instance of an object.
ERROR - Start multi-container app failed.

Can you please advise?

Thread Thread
 
alxizr profile image
alxizr

@reconnect

It seems that the docker compose file is faulty. Do you want to show your file so I can take a look into it.

Thread Thread
 
reconnect profile image
Reconnect

Azure App Services docker-compose.yml Version

version: "3.9"

x-environment: &base_environment
    PYTHONUNBUFFERED: 0
    REDASH_WEB_WORKERS: 4
    REDASH_LOG_LEVEL: "INFO"
    REDASH_RATELIMIT_ENABLED: "false"
    REDASH_REDIS_URL: "redis://redis_server:6379/0"
    REDASH_MAIL_DEFAULT_SENDER: "redash@example.com"
    REDASH_ADDITIONAL_QUERY_RUNNERS: "redash.query_runner.python"

    # do not forget to update this key
    REDASH_DATABASE_URL: "postgresql://<your-admin-username>@<your-managed-server-name>:<your-password>@<your-host>:5432/redash?sslmode=require"


x-base_redash: &base_redash
    environment:
        <<: *base_environment
    image: redash/redash:8.0.2.b37747
    restart: always


services:
    # redis
    redis_server:
        image: redis:alpine
        container_name: redis_server_local
        restart: unless-stopped

    # redash server
    server:
        <<: *base_redash
        command: server
        ports:
        - "5000:5000"
        - "5678:5678"
        depends_on:
        - redis_server

    # redash scheduler
    scheduler:
        <<: *base_redash
        command: scheduler
        depends_on:
        - server
        environment:
        << : *base_environment
        QUEUES: "celery"
        WORKERS_COUNT: 1

    # redash worker 1
    scheduled_worker:
        <<: *base_redash
        command: worker
        depends_on:
        - server
        environment:
        << : *base_environment
        QUEUES: "scheduled_queries"
        WORKERS_COUNT: 1

    # redash worker 2
    adhoc_worker:
        <<: *base_redash
        command: worker
        depends_on:
        - server
        environment:
        << : *base_environment
        QUEUES: "queries"
        WORKERS_COUNT: 2

    # redash worker 3
    scheduled_worker:
        <<: *base_redash
        command: worker
        depends_on:
        - server
        environment:
        << : *base_environment
        QUEUES: "schemas"
        WORKERS_COUNT: 1

    # nginx
    nginx:
        image: redash/nginx:latest
        ports:
        - "8080:80"
        depends_on:
        - server
        links:
        - server:redash
        restart: always
Enter fullscreen mode Exit fullscreen mode
Thread Thread
 
reconnect profile image
Reconnect

sending the file again - made a change in - # redash worker 1 - scheduled_worker: - QUEUES: "schemas,scheduled_queries". As it was giving duplicate key error. but the same errors are still there.

Azure App Services docker-compose.yml Version

version: "3.9"

x-environment: &base_environment
    PYTHONUNBUFFERED: 0
    REDASH_WEB_WORKERS: 4
    REDASH_LOG_LEVEL: "INFO"
    REDASH_RATELIMIT_ENABLED: "false"
    REDASH_REDIS_URL: "redis://redis_server:6379/0"
    REDASH_MAIL_DEFAULT_SENDER: "redash@example.com"
    REDASH_ADDITIONAL_QUERY_RUNNERS: "redash.query_runner.python"
    # do not forget to update this key
    REDASH_DATABASE_URL: "postgresql://<your-admin-username>@<your-managed-server-name>:<your-password>@<your-host>:5432/redash?sslmode=require"

x-base_redash: &base_redash
    environment:
        <<: *base_environment
    image: redash/redash:8.0.2.b37747
    restart: always

services:
    # redis
    redis_server:
        image: redis:alpine
        container_name: redis_server_local
        restart: unless-stopped

    # redash server
    server:
        <<: *base_redash
        command: server
        ports:
        - "5000:5000"
        - "5678:5678"
        depends_on:
        - redis_server

    # redash scheduler
    scheduler:
        <<: *base_redash
        command: scheduler
        depends_on:
        - server
        environment:
        << : *base_environment
        QUEUES: "celery"
        WORKERS_COUNT: 1

    # redash worker 1
    scheduled_worker:
        <<: *base_redash
        command: worker
        depends_on:
        - server
        environment:
        << : *base_environment
        QUEUES: "schemas,scheduled_queries"
        WORKERS_COUNT: 1

    # redash worker 2
    adhoc_worker:
        <<: *base_redash
        command: worker
        depends_on:
        - server
        environment:
        << : *base_environment
        QUEUES: "queries"
        WORKERS_COUNT: 2

    # nginx
    nginx:
        image: redash/nginx:latest
        ports:
        - "8080:80"
        depends_on:
        - server
        links:
        - server:redash
        restart: always
Enter fullscreen mode Exit fullscreen mode
Thread Thread
 
alxizr profile image
alxizr • Edited

something is wrong with the YML file. The indentations are all wrong. You need to remember that the YML format is very sensitive to the the indentations! It takes to space characters (white space) for the indent. As far as i can see, your main issue is with the indentations.

Thread Thread
 
alxizr profile image
alxizr

Everything under this line: x-environment: &base_environment should have a single indent and not double indent

Thread Thread
 
alxizr profile image
alxizr • Edited

for the # redash scheduler service pay attention to the environment key. QUEUES and WORKERS_COUNT are keys of the environment but because of the wrong indents it looks like they are part of the YML file

scheduler:
  <<: *base_redash
  command: scheduler
  depends_on:
    - server
  environment:
    << : *base_environment
    QUEUES: "celery"
    WORKERS_COUNT: 1

Thread Thread
 
alxizr profile image
alxizr • Edited

for the # redash server service pay attention that all the values are set as keys. Under the key ports the - character should be indented.

  server:
    <<: *base_redash
    command: server
    ports:
      - "5000:5000"
      - "5678:5678"
    depends_on:
      - redis_server

Thread Thread
 
alxizr profile image
alxizr • Edited

for the # redash worker 1 & 2 services it is all the same as the # redash scheduler service. Pay attention to all the keys

Thread Thread
 
alxizr profile image
alxizr

for the # nginx service your keys are: image, ports, depends_on, links and restart, everything under each line suppose to be the value, so we need to indent it as well.

  nginx:
    image: redash/nginx:latest
    ports:
      - "8080:80"
    depends_on:
      - server
    links:
      - server:redash
    restart: always

Thread Thread
 
alxizr profile image
alxizr • Edited

yamllint.com/

This is a tool i often use when I need to write a long and complex YML file. I talked about it in my article about YML. There is a link in the first part of this series.

Just copy and paste your YML file content inside this tool's the editor and validate it. In case something is wrong, you will get a notification error.

Thread Thread
 
reconnect profile image
Reconnect

HI, Thanks for helping and suggesting the chnages in the yml file. Made the changes and checked with yamllint.com and validated. But still getting the following error:
ERROR - Exception in multi-container config parsing: Exception: System.NullReferenceException, Msg: Object reference not set to an instance ERROR - Start multi-container app failed.

Do you think there is an alternate approach to deploy using azure ACI: docker.com/blog/how-to-deploy-cont...

Following is the yml file:

docker-compose.yml

version: "3.9"

x-environment: &base_environment
PYTHONUNBUFFERED: 0
REDASH_WEB_WORKERS: 4
REDASH_LOG_LEVEL: "INFO"
REDASH_RATELIMIT_ENABLED: "false"
REDASH_REDIS_URL: "redis://redis_server:6379/0"
REDASH_MAIL_DEFAULT_SENDER: "redash@example.com"
REDASH_ADDITIONAL_QUERY_RUNNERS: "redash.query_runner.python"
REDASH_DATABASE_URL: "MY URL"

x-base_redash: &base_redash
environment:
<<: *base_environment
image: redash/redash:8.0.2.b37747
restart: always

services:
# redis
redis_server:
image: redis:alpine
container_name: redis_server_local
restart: unless-stopped

# redash server
server:
<<: *base_redash
command: server
ports:
- "5000:5000"
- "5678:5678"
depends_on:
- redis_server

# redash scheduler
scheduler:
<<: *base_redash
command: scheduler
depends_on:
- server
environment:
<<: *base_environment
QUEUES: "celery"
WORKERS_COUNT: 1

# redash worker 1
scheduled_worker:
<<: *base_redash
command: worker
depends_on:
- server
environment:
<<: *base_environment
QUEUES: "scheduled_queries,schemas"
WORKERS_COUNT: 1

# redash worker 2
adhoc_worker:
<<: *base_redash
command: worker
depends_on:
- server
environment:
<<: *base_environment
QUEUES: "queries"
WORKERS_COUNT: 2

# nginx - pay attention to the image name
nginx:
image: redash/nginx:latest
ports:
- "8080:80"
depends_on:
- server
links:
- server:redash
restart: always

Thread Thread
 
alxizr profile image
alxizr

regarding the ACI it can be a possible solution but I am not sure because i never tried it with the Redash. My experience with it was that when I did create containers with ACI, all of them should be on the same network so they could communicate.

Thread Thread
 
alxizr profile image
alxizr • Edited

regarding the error, I can not understand where it comes from, so what I will suggest is you isolate every service in the YML file and see when it occurs. You have 6 services and once you find the service that causing the error then we would be able to solve it hopefully

Thread Thread
 
reconnect profile image
Reconnect

hi, we were able to deploy using a single VM node and its working. We got a domain tied to it. After getting the cert for my domain, everything works except have issue with one the web route not working. If you know something about it and can provide some advise and help.

my-website.com doesn't work.

Whereas rest of the following 3 web routes works fine:
my-website.com
my-website.com
my-website.com

Collapse
 
alxizr profile image
alxizr • Edited

@formerlyknownas
@reconnect

I would suggest you first see that there are no firewall rules, custom policies, networking bans of some sorts that could potentially deny your IP address get to the managed instance.

Collapse
 
formerlyknownas profile image
formerlyknownas

Thanks for sharing this! From a DevOps perspective, what has it been like to maintain and host the Redash service? We'd like to continue using it as it provides a lot of value for the company, but I'm not entirely sure if we should go the self hosted route just yet. We currently use their hosted service which is getting phased out in November.

Collapse
 
alxizr profile image
alxizr • Edited

Hi @formerlyknownas , thanks for the question. I can tell you my opinion as i was the lead on this thing. From a DevOps stand point it wasn't as difficult as you might be wondering because all you have is the docker compose file with all the services configured working together. The main issue we had is with the different data connectors whether it is nosql or sql. The amounts of data we put into Redash are huge and we found that as the data sources got bigger the latency grew and Redash took its time in connecting and reading it on an interval. We also noticed that the Elasticsearch connector is not the best. We experienced a lot of disconnections and data loss while reading and transferring the data.

You also need to pay attention to the widgets/dashboards which are dependent on the queries that they in turn are dependent on the connectors. Some widgets/dashboards loaded data or calculated the data slowly, what we did to solve it is actually creating a connector for any dashboard that we saw acting up, this is not the best practice but it did solve our issues.

I hope i was able to answer your question and by all means if you have any more, feel free to ask.

Collapse
 
reconnect profile image
Reconnect

After following all the steps (and do want to mention that its works fine on local machine) getting the following errors on Azure:
ERROR - Exception in multi-container config parsing: Exception: System.NullReferenceException, Msg: Object reference not set to an instance of an object.
ERROR - Start multi-container app failed.

Please advise.