lessons learned from the Qlikview loads the data from DB into qvd files

Qlikview加载数据库里的数据，一般都是通过ODBC和数据库进行连接，然后把数据库里的相应的表里的数据加载到本地的disk folder里，后期就可以直接从用本地的disk folder里的QVD的文件获取数据来做DASHBOARD了。本人一开始加载数据的时候老是会死机，原因是我在加载好一个数据库里的表里数据到本地的QVD文件后，没有及时的从内存里把相应的表给删除掉，导致了多个表都在内存里，而且他们还会通过相同的自动名称进行关联，有时候QLIKVIEW
会自动的产生很多个synthetic的key. 所以在加载数据的时候要及时的把内存里的表给删除掉是个好办法。

下面是我加载数据库里的数据并且把它保存到本来生成QVD 文件的例子：

//Loading the data from DB to local disk folder

BI_CUSTOMER:

LOAD CITY,

"CUSTOMER_ID",

"CUSTOMER_INDUSTRY",

"CUSTOMER_NAME",

"CUSTOMER_NUMBER",

LOB,

"OPERATION_UNIT",

"ORGANIZATION_ID";

SQL SELECT CITY,

"CUSTOMER_ID",

"CUSTOMER_INDUSTRY",

"CUSTOMER_NAME",

"CUSTOMER_NUMBER",

LOB,

"OPERATION_UNIT",

"ORGANIZATION_ID"

FROM DW.dbo."BI_CUSTOMER";

STORE BI_CUSTOMER INTO D:\Qlikview_QVD_NEWDW\DW\BI_CUSTOMER.qvd (QVD);

drop table BI_CUSTOMER;

BI_CUSTOMER_ANLY:

LOAD "AR_OS",

"CUSTOMER_ID",

"CUSTOMER_NAME",

"CUSTOMER_NUMBER",

DSO,

"DSO_YTD",

"MARKET_SEGMENT",

MONTH,

"OPERATING_UNIT",

"OU_ID";

SQL SELECT "AR_OS",

"CUSTOMER_ID",

"CUSTOMER_NAME",

"CUSTOMER_NUMBER",

DSO,

"DSO_YTD",

"MARKET_SEGMENT",

MONTH,

"OPERATING_UNIT",

"OU_ID"

FROM DW.dbo."BI_CUSTOMER_ANLY";

STORE BI_CUSTOMER_ANLY INTO D:\Qlikview_QVD_NEWDW\DW\BI_CUSTOMER_ANLY.qvd (QVD);

drop table BI_CUSTOMER_ANLY;

BI_INV_HEADERS:

LOAD "CUSTOMER_ID",

"CUSTOMER_NAME",

"CUSTOMER_NUMBER",

"INVOICE_DATE",

"INVOICE_NUMBER",

LOB,

"OPERATING_UNIT",

"ORGANIZATION_ID",

"PRIMARY_SALESREP_ID",

"TRANSACTION_ID";

SQL SELECT "CUSTOMER_ID",

"CUSTOMER_NAME",

"CUSTOMER_NUMBER",

"INVOICE_DATE",

"INVOICE_NUMBER",

LOB,

"OPERATING_UNIT",

"ORGANIZATION_ID",

"PRIMARY_SALESREP_ID",

"TRANSACTION_ID"

FROM DW.dbo."BI_INV_HEADERS";

STORE BI_INV_HEADERS INTO D:\Qlikview_QVD_NEWDW\DW\BI_INV_HEADERS.qvd (QVD);

drop table BI_INV_HEADERS;

BI_INV_LINES:

LOAD "CCY_CODE",

"CCY_RATE",

"CCY_RATE_TO_REPORTING",

COGS,

"CONTRIBUTED_MARGIN",

"GROSS_PROFIT",

"INVENTORY_ITEM_ID",

"ITEM_NUMBER",

"MARKET_SEGMENT",

"ORDER_HEADER_ID",

"ORDER_LINE_ID",

"ORDER_NUMBER",

"ORDER_TYPE",

"ORGANIZATION_ID",

"PRIMARY_QUANTITY",

"PRIMARY_UNIT_SELLING_PRICE",

"PRIMARY_UOM_CODE",

"SALES_CHANNEL",

"SALES_OFFICE_CODE",

"SALES_REGION_CODE",

"SALES_TYPE",

"SALESREP_ID",

"TRANSACTION_ID",

"TRX_DATE";

SQL SELECT "CCY_CODE",

"CCY_RATE",

"CCY_RATE_TO_REPORTING",

COGS,

"CONTRIBUTED_MARGIN",

"GROSS_PROFIT",

"INVENTORY_ITEM_ID",

"ITEM_NUMBER",

"MARKET_SEGMENT",

"ORDER_HEADER_ID",

"ORDER_LINE_ID",

"ORDER_NUMBER",

"ORDER_TYPE",

"ORGANIZATION_ID",

"PRIMARY_QUANTITY",

"PRIMARY_UNIT_SELLING_PRICE",

"PRIMARY_UOM_CODE",

"SALES_CHANNEL",

"SALES_OFFICE_CODE",

"SALES_REGION_CODE",

"SALES_TYPE",

"SALESREP_ID",

"TRANSACTION_ID",

"TRX_DATE"

FROM DW.dbo."BI_INV_LINES" WHERE TRX_DATE>‘2012-03-31‘;

STORE BI_INV_LINES INTO D:\Qlikview_QVD_NEWDW\DW\BI_INV_LINES.qvd (QVD);

drop table BI_INV_LINES;

BI_ITEM:

LOAD BRAND,

"CAT_SEG1",

"CAT_SEG2",

COO,

"INVENTORY_ITEM_ID",

"ITEM_CODE",

"ITEM_DESCRIPTION",

LOB,

MANUFACTURER,

MODEL,

"ORGANIZATION_CODE",

"ORGANIZATION_ID",

SERIES,

"UOM_CODE";

SQL SELECT BRAND,

"CAT_SEG1",

"CAT_SEG2",

COO,

"INVENTORY_ITEM_ID",

"ITEM_CODE",

"ITEM_DESCRIPTION",

LOB,

MANUFACTURER,

MODEL,

"ORGANIZATION_CODE",

"ORGANIZATION_ID",

SERIES,

"UOM_CODE"

FROM DW.dbo."BI_ITEM";

STORE BI_ITEM INTO D:\Qlikview_QVD_NEWDW\DW\BI_ITEM.qvd (QVD);

drop table BI_ITEM;

BI_ITEM_ANLY:

LOAD DOS,

"DOS_YTD",

"INVENTORY_ITEM_ID",

"ITEM_NUMBER",

LOB,

MONTH,

"ORGANIZATION_CODE",

"ORGANIZATION_ID";

SQL SELECT DOS,

"DOS_YTD",

"INVENTORY_ITEM_ID",

"ITEM_NUMBER",

LOB,

MONTH,

"ORGANIZATION_CODE",

"ORGANIZATION_ID"

FROM DW.dbo."BI_ITEM_ANLY";

STORE BI_ITEM_ANLY INTO D:\Qlikview_QVD_NEWDW\DW\BI_ITEM_ANLY.qvd (QVD);

drop table BI_ITEM_ANLY;

BI_OU:

LOAD "ccy_code",

"organization_id",

"ou_code",

"ou_name",

"short_name";

SQL SELECT "ccy_code",

"organization_id",

"ou_code",

"ou_name",

"short_name"

FROM DW.dbo."BI_OU";

STORE BI_OU INTO D:\Qlikview_QVD_NEWDW\DW\BI_OU.qvd (QVD);

drop table BI_OU;

BI_SALESREPS:

LOAD "END_DATE_ACTIVE",

"SALES_OFFICE_CODE",

"SALES_OFFICE_NAME",

"SALES_REGION_CODE",

"SALES_REGION_NAME",

"SALESREP_ID",

"SALESREP_NAME",

"SALESREP_NUMBER";

SQL SELECT "END_DATE_ACTIVE",

"SALES_OFFICE_CODE",

"SALES_OFFICE_NAME",

"SALES_REGION_CODE",

"SALES_REGION_NAME",

"SALESREP_ID",

"SALESREP_NAME",

"SALESREP_NUMBER"

FROM DW.dbo."BI_SALESREPS";

STORE BI_SALESREPS INTO D:\Qlikview_QVD_NEWDW\DW\BI_SALESREPS.qvd (QVD);

drop table BI_SALESREPS;

BI_SALESREPS_ANLY:

LOAD "AR_OS",

DSO,

"DSO_YTD",

MONTH,

"OPERATING_UNIT",

"OU_ID",

"SALESREP_ID",

"SALESREP_NAME",

"SALESREP_NUMBER";

SQL SELECT "AR_OS",

DSO,

"DSO_YTD",

MONTH,

"OPERATING_UNIT",

"OU_ID",

"SALESREP_ID",

"SALESREP_NAME",

"SALESREP_NUMBER"

FROM DW.dbo."BI_SALESREPS_ANLY";

STORE BI_SALESREPS_ANLY INTO D:\Qlikview_QVD_NEWDW\DW\BI_SALESREPS_ANLY.qvd (QVD);

drop table BI_SALESREPS_ANLY;

从上面可以看出，我加载完一个表后就把此表从内存中删除掉了，这样会释放出内存给后面的表来使用。



时间： 2024-10-08 15:27:51

lessons learned from the Qlikview loads the data from DB into qvd files的相关文章

Lessons Learned from Developing a Data Product

Lessons Learned from Developing a Data Product For an assignment I was asked to develop a visual ‘data product’ that informed decisions on video game ratings taking as an indicator their ranking on the MetaCritic site. I decided to use RStudio’s Shin

Lessons learned developing a practical large scale machine learning system

原文:http://googleresearch.blogspot.jp/2010/04/lessons-learned-developing-practical.html Lessons learned developing a practical large scale machine learning system Tuesday, April 06, 2010 Posted by Simon Tong, Google Research When faced with a hard pre

Lessons learned from manually classifying CIFAR-10

Lessons learned from manually classifying CIFAR-10 Apr 27, 2011 CIFAR-10 Note, this post is from 2011 and slightly outdated in some places. Statistics. CIFAR-10 consists of 50,000 training images, all of them in 1 of 10 categories (displayed left). T

翻译 | Improving Distributional Similarity with Lessons Learned from Word Embeddings

翻译 | Improving Distributional Similarity with Lessons Learned from Word Embeddings 叶娜老师说:"读懂论文的最好方法是翻译它".我认为这是很好的科研训练,更加适合一个陌生领域的探索.因为论文读不懂,我总结无非是因为这个领域不熟悉.如果是自己熟悉的领域,那么读起来肯定会比较顺畅. 原文 摘要 [1] Recent trends suggest that neural-network-inspired wor

数据库内存泄漏——A SQLiteConnection object for database '/data/data/.../databases/....db' was leaked!

详细异常: A SQLiteConnection object for database '/data/data/.../database/....db' was leaked!  Please fix your application to end transactions in progress properly and to close the database when it is no longer needed 明显是数据库操作异常,数据库对象被锁,明确告诉你对象长久不用需要关闭.

A SQLiteConnection object for database '/data/data/.../databases/....db' was leaked!

详细异常: A SQLiteConnection object for database '/data/data/.../databases/....db' was leaked!  Please fix your application to end transactions in progress properly and to close the database when it is no longer needed 明显是数据库操作异常,数据库对象被锁,明确告诉你对象长久不用需要关闭.

Lessons Learned 1（敏捷项目中的变更影响分析）

问题/现象: 业务信息流转的某些环节,会向相关人员发送通知邮件,邮件中附带有链接,供相关人员进入察看或处理业务.客户要求邮件中的链接,需要进行限制,只有特定人员才能进入处理或察看.总管想了想,应道没问题,不一会儿就改好了,在业务信息的查询方法中添加了限制——非处理人不得进入.测试这边,忙得脚不沾地,一人扛了两个项目的测试,但还是按照预先设计的测试用例,对该修改进行了测试,测试结果ok,非处理人通过邮件链接进入后,确实提示了“你没有权限,翻滚着离去吧”. 当晚发布生产后,客户一封邮件甩过来:管理员

Insert excel data into DB

using System;using System.Collections.Generic;using System.Linq;using System.Web;using System.Web.UI;using System.Web.UI.WebControls;using System.Data;using System.Data.OleDb;using System.Configuration;using System.Data.SqlClient; namespace InExcelOu

以Excel 作为Data Source，将data导入db

将Excel作为数据源,将数据导入db,是SSIS的一个简单的应用,下图是示例Excel,数据列是code和name 第一部分,Excel中的数据类型是数值类型 1,使用SSDT创建一个package,创建Excel data source component,SSDT会在Connection Managers中创建一个Excel的connection 由于示例Excel的首行是列名,所以需要勾选"First row has column names",Excel connection