0
Posted on Tuesday, February 14, 2017 by 醉·醉·鱼 and labeled under ,
之前介绍过通过Extended Event观察锁,这次介绍通过dm_tran_locks来观察锁。



BEGIN TRANSACTION

DELETE TOP (1) p FROM tbl p 

EXEC ##TranLocks

ROLLBACK TRANSACTION



##TranLocks SP

CREATE PROCEDURE ##TranLocks
    @objectName NVARCHAR(255) = '%',
    @requestMode VARCHAR(10) = '%',
    @processID INT = @@SPID,
    @ignoreSharedDatabaseLocks BIT = 1
AS
SELECT 
    DB_NAME() AS database_name,
    CASE 
        WHEN dt1.resource_type = 'OBJECT' THEN object_name(dt1.resource_associated_entity_id, resource_database_id) 
        WHEN dt1.resource_type IN('KEY', 'PAGE') THEN object_name(pa1.object_id) 
        ELSE dt1.resource_description
        END AS object_name,
    in1.name AS index_name,
    pa1.partition_number,
    dt1.resource_type,
    dt1.request_mode,
    COUNT(*) AS locks,
    dt1.request_owner_id,
    dt1.request_session_id
FROM
    sys.dm_tran_locks dt1
    LEFT OUTER JOIN sys.partitions pa1 ON(dt1.resource_associated_entity_id = pa1.hobt_id)
    LEFT OUTER JOIN sys.indexes in1 ON(pa1.object_id = in1.object_id AND pa1.index_id = in1.index_id)
WHERE 
    (request_session_id = @processID OR @processID = 0) 
    AND request_mode LIKE @requestMode
    AND(NOT(dt1.resource_type = 'DATABASE' AND dt1.request_mode = 'S') OR @ignoreSharedDatabaseLocks = 0)
    AND dt1.resource_database_id = DB_ID()
    AND 
        CASE 
            WHEN dt1.resource_type = 'OBJECT' THEN object_name(dt1.resource_associated_entity_id, resource_database_id) 
            WHEN dt1.resource_type IN('KEY', 'PAGE') THEN object_name(pa1.object_id) 
            ELSE dt1.resource_description
            END LIKE @objectName
GROUP BY
    dt1.resource_type,
    CASE 
        WHEN dt1.resource_type = 'OBJECT' THEN object_name(dt1.resource_associated_entity_id, resource_database_id) 
        WHEN dt1.resource_type IN('KEY', 'PAGE') THEN object_name(pa1.object_id) 
        ELSE dt1.resource_description
        END,
    in1.name,
    pa1.partition_number,
    dt1.request_mode,
    dt1.request_session_id,
    dt1.request_owner_id
ORDER BY database_name, object_name, index_name, partition_number, request_mode, locks;
GO

0
Posted on Tuesday, February 14, 2017 by 醉·醉·鱼 and labeled under ,
直接上测试脚本。LOB可以存储VARCHAR(MAX), TEXT, IMAGE这些数据类型。但是各个类型的behavior又不太一样。

和OVERFLOW DATA 一样,数据页会有一个pointer指向LOB root structure,在LOB root structure上,又会分别指向其他LOB data page。如果数据超过32KB,LOB root structure又会引入中间层,像INDEX中的B tree一样。




NVARCHAR(MAX)

IF OBJECT_ID('LobData') IS NOT NULL
    DROP TABLE dbo.LobData

create table dbo.LobData
(
    ID int not null,
    Col1 nvarchar(max) null
);

insert into dbo.LobData(ID, Col1) 
values (1, replicate(convert(varchar(max),'a'),16000));


SELECT allocated_page_file_id as PageFID, allocated_page_page_id as PagePID,
       object_id as ObjectID, partition_id AS PartitionID,
       allocation_unit_type_desc as AU_type, page_type as PageType
FROM sys.dm_db_database_page_allocations(db_id('test_db'), object_id('LobData'),
                                          null, null, 'DETAILED');
GO

DBCC TRACEON (3604); 
GO 
DBCC PAGE ('test_db', 1, 623876, 3); 
GO


当Column为NVARCHAR(MAX)且16000字符串(实际长度为32000)的时候,SQL server不会单独开辟一个新的page去存储LOB root structure。相反,LOB pointer会存在当前page。1个IN_ROW数据页,4个LOB 数据页。指针留在第一个数据页。

DBCC PAGE第一个数据页,可以看到BLOB Inline Root。其中,LOB pointer和ROW_OVERFLOW的pointer类似,如下


  1. 04 00000001 00000020 1c0000 metadata
  2. 68 1f0000 length
  3. 01 850900 page number
  4. 01 00 file number
  5. 0000 slow number

page number按照前一个文章的方法,实际应该是0x98501,即623873。同理可以拿到剩下3个pointer。

d0 3e000002 850900
01 000000
38 5e000003 850900
01 000000
00 7d000037 520900
01 000000




TEXT


可能是对deprecated 数据类型支持不好,对于TEXT直接就开辟一个新的page为LOB root structure。DBCC PAGE LOB root structure,可以看到LOB pointer。

VARCHAR(MAX)

和NVARCHAR(MAX)一样的行为,当数据比较短的时候,存为BLOB Inline Root。当数据更短的时候,直接就存在当前IN_ROW page。


IF OBJECT_ID('LobData') IS NOT NULL
    DROP TABLE dbo.LobData

create table dbo.LobData
(
    ID int not null,
    Col1 varchar(max) null
);

insert into dbo.LobData(ID, Col1) 
values (1, replicate(convert(varchar(max),'a'),400));

在这个例子中,只有两个page。一个是IAM,另外一个就是数据页,没有单独的LOB page。
DBCC page数据页,可以看到BLOB Inline Data。



参考:

  1. http://aboutsqlserver.com/2013/11/05/sql-server-storage-engine-lob-storage/
  2. http://improve.dk/what-is-the-size-of-the-lob-pointer-for-max-types-like-varchar-varbinary-etc/





0
Posted on Monday, February 13, 2017 by 醉·醉·鱼 and labeled under ,

SQL Server有IN_ROW,ROW_OVERFLOW和LOB 3种page。

首先,来看看ROW_OVERFLOW是怎么存储的吧。

首先创建一个Table,插入一下数据。ID和Col1会存储在第一个page上,即IN_ROW page。


IF OBJECT_ID('RowOverflow') IS NOT NULL
    DROP TABLE RowOverflow;
GO
create table dbo.RowOverflow 
( 
    ID int not null, 
    Col1 varchar(8000) null, 
    Col2 varchar(8000) null 
);

insert into dbo.RowOverflow(ID, Col1, Col2) 
values (1,replicate('a',8000),replicate('b',8000));


验证一下表的数据页
SELECT allocated_page_file_id as PageFID, allocated_page_page_id as PagePID,
       object_id as ObjectID, partition_id AS PartitionID,
       allocation_unit_type_desc as AU_type, page_type as PageType
FROM sys.dm_db_database_page_allocations(db_id('test_db'), object_id('RowOverflow'),
                                          null, null, 'DETAILED');
GO

结果如图。一共有4个page。两个IAM,一个数据页,另外一个LOB或者Overflow page。

执行DBCC PAGE,可以看到第一页的存储情况。

DBCC TRACEON (3604); 
GO 
DBCC PAGE ('test_db', 1, 610871, 3); 
GO



数据的最后一共有24bytes,这些都是字节交换的,所以需要倒着读。

  1. 02000000 01000000 29000000 401f0000 是metadata attribute。
  2. 31520900 是page number
  3. 0001 是file number
  4. 0000 是slot number

在Kalen写的SQL SERVER 2012 internals里面提到了metadata attribute。

The first 16 bytes of a row-overflow pointer

Bytes
Hex value
Decimal value
Meaning
0
0x02
2
Type of special field: 1 = LOB2 = overflow
1–2
0x0000
0
Level in the B-tree (always 0 for overflow)
3
0x00
0
Unused
4–7
0x00000001
1
Sequence: a value used by optimistic concurrency control for cursors that increases every time a LOB or overflow column is updated
8–11
0x00007fc3
32707
Timestamp: a random value used by DBCC CHECKTABLE that remains unchanged during the lifetime of each LOB or overflow column
12–15
0x00000834
2100
Length

31520900应该读作0X00095231,换做十进制的610865。


继续往下看,你可以看到Col2的RowId记录的page number。这个也印证了我们从dm_db_database_page_allocations里看到的情况。
来看看ROW_OVERFLOW page的情况。就可以看到Col2的数据了。


DBCC PAGE ('test_db', 1, 610871, 3); 
GO


参考
  1. http://aboutsqlserver.com/2013/11/05/sql-server-storage-engine-lob-storage/
  2. https://www.microsoftpressstore.com/articles/article.aspx?p=2225060
0
Posted on Sunday, January 22, 2017 by 醉·醉·鱼 and labeled under
总是不喜欢UI,点得慢死了~


USE Master; 
GO  
SET NOCOUNT ON 

-- 1 - Variable declaration 
DECLARE @dbName sysname 
DECLARE @backupPath NVARCHAR(500) 
DECLARE @dbDataPath NVARCHAR(500)
DECLARE @cmd NVARCHAR(500) 
DECLARE @fileList TABLE (backupFile NVARCHAR(255)) 
DECLARE @lastFullBackup NVARCHAR(500) 
DECLARE @backupFile NVARCHAR(500) 
DECLARE @NoExec bit
DECLARE @debug bit


-- 2 - Initialize variables 
SET @dbName = 'KEY_WORD_IN_YOUR_BACKUP_FILE' 
SET @backupPath = 'D:\' 
SET @dbDataPath = 'D:\Data\'
SET @NoExec = 1
SET @debug = 1

-- 3 - get list of files 
SET @cmd = 'DIR /b ' + @backupPath 

INSERT INTO @fileList(backupFile) 
EXEC master.sys.xp_cmdshell @cmd 

-- 4 - Find latest full backup 
SELECT @lastFullBackup = MAX(backupFile)  
FROM @fileList  
WHERE backupFile LIKE '%.BAK'  
   AND backupFile LIKE 'Servlet.' + @dbName + '%' 

SET @cmd = 'RESTORE DATABASE ' + @dbName + ' FROM DISK = '''  
       + @backupPath + @lastFullBackup + '''' + CHAR(10) + 'WITH RECOVERY, REPLACE,'  + CHAR(10)
  + 'MOVE N''RECNETSTARTUP_dat'' TO N''' + @dbDataPath + @dbName + '.mdf'','   + CHAR(10)
  + 'MOVE N''RECNETSTARTUP_log'' TO N''' + @dbDataPath + @dbName + '_log.LDF'',  NOUNLOAD,  STATS = 5'

IF @debug = 1
PRINT @cmd 
IF @NoExec <> 1
EXEC (@cmd)




0
Posted on Monday, January 16, 2017 by 醉·醉·鱼 and labeled under
可能大家都习惯了SELECT * FROM TABLE,或者SELECT DATA FROM TABLE WHERE ID = 1。一般来说,无非就是INDEX SEEK,或者CLUSTERED INDEX SCAN。但下面这个例子,却两者都不是。


IF OBJECT_ID('DBO.T1') IS NOT NULL
    DROP TABLE t1

CREATE TABLE T1(
    ID BIGINT,
    ANOTHER_ID INT,
    DATA VARCHAR(200),
    CONSTRAINT PK_T1 PRIMARY KEY CLUSTERED (ID)
)

INSERT INTO T1(ID, ANOTHER_ID,DATA)
SELECT n,
    n%4, 
    CASE WHEN n % 4 = 1 THEN 'pHoEnIx'
                    WHEN n % 4 = 2 THEN 'eRiK'
                    WHEN n % 4 = 3 THEN 'dBe'
                    ELSE 'Papa'
                    END
FROM dbo.getnums(50000)

CREATE NONCLUSTERED INDEX idx_t1_another_id ON t1(another_id)

SELECT ID FROM T1
SELECT DATA FROM T1



对于第二个SELECT query,很明显,DATA不被包含在non-clustered index里面,所以只能够用CLUSTERED INDEX SCAN。但是第一个SELECT query,却是INDEX SCAN。WHY?是错误的执行计划?不是。
事实是,很明显non-clustered index占用的空间比clustered index小,而且non-clustered index的最后一个节点就是ID,不用它用谁。此外,如果一个表上面有多个non-clustered index,SQL SERVER会用INDEX占用最小的INDEX。



如果一切这么简单就好,这样的query单独列出来可能稍微仔细看就发现问题了。继续上面的SQL script


CREATE TABLE T2(
    ID INT
)

INSERT INTO T2(ID)
SELECT n*4
FROM dbo.getnums(50000)

DELETE FROM T2 WHERE NOT EXISTS(SELECT 1 FROM T1 WHERE T1.ID = t2.ID)



当T2的数据级和T1的数据级差不多的时候,SQL SERVER就不会SCAN T2再去T1里面进行CLUSTERED INDEX SEEK。相反,SQL SERVER会对两个表都进行SCAN,再HASH MATCH。这里很隐晦地把SELECT ID FROM TABLE转换成SELECT 1 FROM TABLE WHERE ID ...。满心以为这个很明显的是CLUSTERED INDEX SEEK啊, WHERE ID = 啊。等你看执行计划的时候,你就傻眼了。啥?!INDEX SCAN??


一旦是INDEX SCAN,一般都是PAGE级别的锁,放在non-clustered index上,一不留神就会导致长时间BLOCKING。
0
Posted on Monday, November 21, 2016 by 醉·醉·鱼 and labeled under , ,
貌似EI Capitan和以前的版本的安装有些差别,记录一下。大体来说,你需要安装下面3个部分。

  • Oracle Instant Client
  • ruby-oci8 gem
  • activerecord-oracle_enhanced-adapter gem

安装Oracle Instant Client

去Oracle官网下载下面几个包,并按照 http://www.oracle.com/technetwork/topics/intel-macsoft-096467.html
  1. instantclient-basic-macos.x64-12.1.0.2.0.zip
  2. instantclient-sqlplus-macos.x64-12.1.0.2.0.zip
  3. instantclient-sdk-macos.x64-12.1.0.2.0.zip
解压到/opt/oracle/instantclient_12_1

cd ~

unzip instantclient-basic-macos.x64-12.1.0.2.0.zip
unzip instantclient-sqlplus-macos.x64-12.1.0.2.0.zip
unzip instantclient-sdk-macos.x64-12.1.0.2.0.zip

3. 创建link


cd /opt/oracle/instantclient_12_1
ln -s libclntsh.dylib.12.1 libclntsh.dylib

Note: OCCI programs will additionally need:


ln -s libocci.dylib.12.1 libocci.dylib

4. 配置PATH


export ORACLE_HOME=/opt/oracle/instantclient_12_1
export OCI_DIR=/opt/oracle/instantclient_12_1
export PATH=$ORACLE_HOME:$PATH
export TNS_ADMIN=$HOME
export NLS_LANG="AMERICAN_AMERICA.UTF8"

安装Gem

gem install 'ruby-oci8' -v '~> 2.1.0'
gem install 'activerecord-oracle_enhanced-adapter' -v '~> 1.5.0'


测试


ActiveRecord::Base.establish_connection(
    :adapter => "oracle_enhanced",
    :database => "database",
    :username => "username",
    :password => "password")

    cursor = ActiveRecord::Base.connection.execute("SELECT 1 n FROM table")

    # query data
    result_data = []

    result_data << cursor.column_metadata.map { |e| e.name }

    while row = cursor.fetch
        result_data << row
    end

引用
  1. http://stackoverflow.com/questions/36811473/ruby-oci8-installation-error-in-mac-el-capitan
  2. https://craig.io/setting-up-a-rails-development-environment-with-oracle/
0
Posted on Thursday, November 17, 2016 by 醉·醉·鱼 and labeled under


  1. ODBC。Open Database Connectivity,是很老的一个数据库连接API。现在基本上没有用了。
  2. 微软后来开发了OLE DB,算是ODBC的替代品,同时支持更多的数据源,比如spreadsheets
  3. ADO.NET是基于.NET framework来连接关系型和非关系型数据库。看上去像ADO的进化版,实际上完全是全新的东西。
  4. JDBC。同ODBC,不过是给JAVA用的。
  5. JDBC有两款driver,一个是微软自己开发的sqljdbc4,另外一个是jtds。前者不支持NamedPipe。
  6. 可以通过
    jdbc:jtds:sqlserver://./DatabaseName;instance=LOCALDB#88893A09;namedPipe=true 连接namedPipe
  7. jtds是基于FreeTds。
  8. ruby下面tiny_tds也是基于FreeTds的。