Analytics Platform System (APS) AU2 released

The Analytics Platform System (APS), which is a renaming of the Parallel Data Warehouse (PDW), has recently released an appliance update (AU2), which is sort of like a service pack, except that it includes many new features.  These appliance updates are made available frequently, about every 3-4 months.  Below is what is new in this release:

TSQL Compatibility improvements to reduce migration friction from SQL SMP

  • Adds full support for user defined schemas. This is one of the most requested features from customers migrating from SQL Server (CREATE SCHEMA, ALTER USER, GRANT/DENY, etc.)
  • Improves statistics collection capability and supportability by introducing filtered statistics, ability to control the sample size for sampled statistics, and support for DBCC SHOW_STATISTICS and STATS_DATE intrinsics
  • Adds support for a set of 51 mathematical, string ODBC, date/time, data and type analytic built in functions:
    ACOS, ATN2, COT, DEGREES, PI, RADIANS, ASCII, CHAR, CONCAT, DIFFERENCE, NCHAR, REVERSE, SPACE, STUFF, UNICODE, TERTIARY_WEIGHTS, PARSENAME, BIT_LENGTH, CURRENT_DATE, CURRENT_TIME, DAYNAME, DAYOFMONTH, DAYOFWEEK, HOUR, MINUTE, SECOND, MONTHNAME, QUARTER, WEEK, DATEFROMPARTS, DATETIME2FROMPARTS, DATETIMEFROMPARTS, DATETIMEOFFSETFROMPARTS, SMALLDATETIMEFROMPARTS, TIMEFROMPARTS, SYSDATETIME, SYSDATETIMEOFFSET, SYSUTCDATETIME, ISDATE, DATENAME, GETUTCDATE, EOMONTH, SWITCHOFFSET, TODATETIMEOFFSET, HASHBYTES, DATALENGTH, TYPE_ID, TYPE_NAME, TYPEPROPERTY, SQL_VARIANT_PROPERTY
  • Enables broader set of 3rd party ISV tools such as SAS and Tableau, as well as unblocking scenarios around managing data and analytical functions within the appliance

HDI Region Support for HDP 2.0 and Yarn

  • Enables users to write custom apps based on new data processing models (beyond MapReduce), enabling new scenarios such as interactive/ad hoc querying or search
  • Improves processing on the same hardware with predictable performance and better quality of service
  • On boards SCOM management pack for HDI, enabling System Center management of PDW+HDI
  • Updates the Microsoft .NET SDK for Hadoop to add value for both APS and Azure HDInsight clusters

Polybase HDP 2.0 and Query Hint Support

  • Enables all Polybase scenarios to work against both HDP 2.0 on Windows and Linux, pushing computation through the Yarn framework
  • Enables query hint support for Polybase push-down, enabling ability to force or disable push-down on a per-query basis

Install, Upgrade, and Servicing Improvements

  • Reduces end-to-end install time from 12 to under 9 hours (single scale unit)
  • Reduces end-to-end install time for virtual appliances from 7.5 to 4.5 hours
  • Reduces end-to-end servicing and patching time from 2 to 1 hour
  • Improves supportability with fast-fail and simplified troubleshooting steps
  • Improves performance by performing parallel and group patching.

Appliance Heartbeat Monitoring

  • Collects critical alerts and appliance heartbeats, storing appliance health data in Azure

Self-host Appliance and Customer Workload

  • On boards key first party workloads by creating APS-based internal data warehouse

Engineering Efficiency Improvements

  • Introduces train model, reducing release cycle to 120 days end-to-end
  • Enables more frequent test passes to reduce bug escape
  • Completely automates virtual appliance allocation

TFS and MSBuild Migration

  • Migrates codebase to Team Foundation Server (TFS) under PDW_Main
  • Converts codebase to MSBuild
Posted in PDW/APS, SQLServerPedia Syndication | 1 Comment

Non-obvious APS/PDW benefits

The Analytics Platform System (APS), which is a renaming of the Parallel Data Warehouse (PDW), has a lot of obvious benefits, which I discuss here.  For those of you who find your database is getting too big, or becoming too slow, or you need to integrate non-relational data, check out APS, Microsoft’s MPP solution.

But there are a lot of non-obvious benefits to using APS that I have listed below:

  • APS is a hardware platform that removes roadblocks for future needs. Be proactive instead of reactive!
  • Numerous capabilities (i.e. PolyBase, mixed workload support, scalability, HDInsight) to get you thinking about better ways to use your data
  • Much quicker development time due to speed of execution
  • Allows removal of ETL complexity (temp tables, aggregation tables, data marts, other band aids and workarounds)
  • Permits elimination of SSAS cubes or switch to ROLAP mode (so real-time data and no cube processing time)
  • Reduction or elimination of nightly maintenance windows. Instead use intra-day batch cycles
  • Tuning, redesigning ETL, upgrading hardware (more memory, Fusion IO cards), etc., for SMP to get 20-50% improvement versus 20-50x improvement with MPP
  • SMP is optimized for OLTP while APS is optimized for data warehouses
  • Allows a clear path to do predictive analytics via tools like Azure ML by having the disk space and processing power
  • Don’t think of it as just a solution for a large data warehouse but rather for any size warehouse that needs faster query performance
  • Faster query performance allows for adding more parameters to reports that also can be run real-time (instead of at night with fixed parameters) so business users can ask more sophisticated questions
  • Have the space and performance to consolidate all your various data warehouses and data marts to one place
Posted in PDW/APS, SQLServerPedia Syndication | 3 Comments

IaaS, PaaS, and SaaS explained

You might be reading a lot about Cloud computing and see three acronyms frequently: IaaS, PaaS, Saas.  Cloud providers offer their services according to these three fundamental models:

Infrastructure-as-a-service (IaaS)

This is the most basic model which is essentially your virtual machines in a cloud data center.  You set up, configure, and manage VMs that run in the data center infrastructure, and you put whatever you want on them.  A hypervisor such as Hyper-V runs the virtual machines as guests.  Pools of hypervisors installed at a data center can support large numbers of virtual machines and the ability to scale services up and down according to customers’ varying requirements.  Windows Azure, Hortonworks Data Platform, Amazon Elastic Compute Cloud (EC2)Rackspace, and Google Compute Engine are the most popular examples.

Traits of IaaS:

  • You Build/Upload Virtual Machines to a DC on the Internet – e.g. Windows Azure
  • You PAY for time/resources used and the software in your VM’s
  • Your virtual machines RUN on hardware shared with other organizations
  • You manage ALL aspects of the software stack inside your virtual machines
  • You perform OS updates and manage runtime and middleware
  • VM’s can be moved to/from the Cloud and your own data center
  • App development is unchanged

Platform-as-a-service (PaaS)

With PaaS, a provider delivers a computing platform, typically including operating system, programming language execution environment, database, and web server.  You don’t have to worry about OS updates or managing runtime and middleware.  The provider manages the hardware and software infrastructure and you just use the service.  It is usually a layer on top of IaaS.  Examples are Microsoft Azure SQL Database, HDInsight, AWS Elastic Beanstalk, Windows Azure BLOB Storage, and Google App Engine.

Using a Windows Azure BLOB Storage example:

  • You SUBSCRIBE to the service and create a unique name
  • You GIVE Blobs(Files) to the Storage Service – simple API or REST
  • The service provides resilience and scale, you don’t have to.
  • You ask for them back – you don’t care or know where they really are (which VM’s)
  • The service and the fabric controller make sure your data is stored so there is no single point of failure
  • You pay for the amount of storage you use – the service manages everything
  • The service can also geo-replicate, provide disaster recovery

Software-as-a-service (SaaS)

With SaaS, users are provided access to application software and databases. Cloud providers manage the infrastructure and platforms that run the applications.  SaaS is sometimes referred to as “on-demand software”.  Google Apps (which includes GMail), Salesforce, and Microsoft Office 365 are good examples.

Traits of SaaS:

  • Complete apps you use
  • Subscribe, on-board, normally pay for the # of users who use the app
  • No access to underlying platform
  • Software may support some customizations
  • Shared hardware, platform and finished software across multiple customers
  • A layer on top of PaaS

More info:

Windows Azure – Write, Run or Use Software

But what can I *do* with Windows Azure? Create (Free) Websites and Applications

IaaS, PaaS and SaaS Terms Clearly Explained and Defined

Cloud Jargon Unwound: Distinguishing SaaS, IaaS and PaaS

What Is Cloud Computing?

Cloud Service Models (IaaS, SaaS, PaaS) + How Microsoft Office 365, Azure Fit In

Microsoft Azure for Enterprises

Cloud Models (IaaS, PaaS, SaaS) explained with examples

Posted in SQLServerPedia Syndication | Leave a comment

Presentation slides for Modern Data Warehousing

Thanks to everyone who attended my session “Modern Data Warehousing” at the Central New Jersey SQL User Group yesterday.  The abstract for my session is below.  I hope you enjoyed it!

Here is the PowerPoint presentation: Modern Data Warehousing

Modern Data Warehousing

The traditional data warehouse has served us well for many years, but new trends are causing it to break in four different ways: data growth, fast query expectations from users, non-relational/unstructured data, and cloud-born data.  How can you prevent this from happening?  Enter the modern data warehouse, which is able to handle and excel with these new trends.  It handles all types of data (Hadoop), provides a way to easily interface with all these types of data (PolyBase), and can handle “big data” and provide fast queries.  Is there one appliance that can support this modern data warehouse?  Yes!  It is the Analytics Platform System (APS) from Microsoft (formally called the Parallel Data Warehouse or PDW), which is a Massively Parallel Processing (MPP) appliance that has been recently updated (v2 AU1).  In this session I will dig into the details of the modern data warehouse and APS.  I will give an overview of the APS hardware and software architecture, identify what makes APS different, and demonstrate the increased performance.  In addition I will discuss how Hadoop, HDInsight, and PolyBase fit into this new modern data warehouse.

Posted in Data warehouse, Presentation, Session, SQLServerPedia Syndication | 1 Comment

Modern Data Warehousing Presentation

I will be presenting the session “Modern Data Warehousing” tomorrow (Wednesday, August 13th) at the Central New Jersey SQL User Group at 6:30pm.  The abstract for my session is below.  I hope you can make it!

Modern Data Warehousing

The traditional data warehouse has served us well for many years, but new trends are causing it to break in four different ways: data growth, fast query expectations from users, non-relational/unstructured data, and cloud-born data.  How can you prevent this from happening?  Enter the modern data warehouse, which is able to handle and excel with these new trends.  It handles all types of data (Hadoop), provides a way to easily interface with all these types of data (PolyBase), and can handle “big data” and provide fast queries.  Is there one appliance that can support this modern data warehouse?  Yes!  It is the Analytics Platform System (APS) from Microsoft (formally called the Parallel Data Warehouse or PDW), which is a Massively Parallel Processing (MPP) appliance that has been recently updated (v2 AU1).  In this session I will dig into the details of the modern data warehouse and APS.  I will give an overview of the APS hardware and software architecture, identify what makes APS different, and demonstrate the increased performance.  In addition I will discuss how Hadoop, HDInsight, and PolyBase fit into this new modern data warehouse.

Posted in Data warehouse, Presentation, Session, SQLServerPedia Syndication | Leave a comment

24 Hours of PASS videos online

All the videos for the sessions at 24 Hours of PASS (Spring 2014) are available for free.  Check the session schedule for a complete list.  Also, many of the PASS virtual chapters have recordings of their meetings online.  Enjoy!

Posted in PASS, SQL Server, SQLServerPedia Syndication, Videos | Leave a comment

SQL Server Agent job steps vs SSIS

When doing ETL, you have the choice of using T-SQL or SSIS (see When to use T-SQL or SSIS for ETL).  If you decide T-SQL is the way to go and you just want to execute a bunch of T-SQL statements (individually or within a stored procedure), it’s still a good idea to wrap them in SSIS Execute SQL Tasks because you can use logging, auditing and error handling that SSIS provides that T-SQL does not.  You can also easily run SSIS Execute SQL Tasks in parallel, so if those tasks are calling stored procedures that means you are able to run stored procedures in parallel.  Other benefits for using SSIS instead of a SQL Server Agent job include:

  • The ability to use a project data connection manager, so if the connection info changes you only need to change it in one spot
  • You can create checkpoints for restarting
  • You can add logic to check if packages have run by querying the status log (SSISDB catalog) instead of manually looking at the SQL Server agent job steps
  • You can do reporting off of the auditing info you capture
  • You can use select statements against the SSIS history for analysis (history stored in SSISDB catalog), which you don’t have for job steps in SQL Server agent
  • Ease of maintenance (but depends on knowledge of SSIS vs knowledge of SQL Server)

 

Posted in SQLServerPedia Syndication, SSIS | 2 Comments

IT books that should be on your shelf

I have a lot of books on my shelves dealing with business intelligence, data warehousing, master data management, and consulting.  Below are my favorites:

Data Warehouse Books:

Ralph Kimball Books

Building the Data Warehouse

Corporate Information Factory

DW 2.0

Mastering Data Warehouse Design

Data Warehouse from Architecture to Implementation

Microsoft SQL Server 2012 Internals

Star Schema: The complete reference

Microsoft Big Data Solutions

Business Intelligence Books:

Expert Cube Development with SSAS Multidimensional Models

Pro SQL Server 2012 BI Solutions

Business Intelligence Competency Centers

Data Mining with Microsoft SQL Server 2008

Performance Dashboards

How To Measure Anything

Dimensional Modeling: In a Business Intelligence Environment

The Balanced Scorecard

Information Dashboard Design

Microsoft SQL Server 2014 Business Intelligence Development

Reporting with Microsoft SQL Server 2012 (shameless plug of my book)

Master Data Management Books:

Master Data Management

Microsoft SQL Server 2012 Master Data Services

Data Quality: The Accuracy Dimension

Consulting Books:

How to win friends and influence people

Influence: Science and practice

The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully

More Secrets of Consulting

The Rational Guide to: IT Consulting

The Nomadic Developer

Agile Project Management With Scrum

Posted in Career, SQLServerPedia Syndication | 1 Comment

PASS BA Conference 2014: Sessions Recordings Now Available

All PASS Business Analytics Conference 2014 recordings are now available.  If you purchased the online sessions, to view, log on to your myPASS account, enter your activation code, and watch the sessions from myRecordings.  You can also purchase all recordings on a USB drive.  While there, check out mine: Building an Effective Data Warehouse Architecture

Posted in PASS, SQLServerPedia Syndication, Videos | Leave a comment

Presentation slides for Modern Data Warehousing

Thanks to everyone who attended my session “Modern Data Warehousing” for Pragmatic Works last Thursday.  The abstract for my session is below and the recording is available here.  I hope you enjoyed it.

Here is the PowerPoint presentation: Modern Data Warehousing

Modern Data Warehousing

The traditional data warehouse has served us well for many years, but new trends are causing it to break in four different ways: data growth, fast query expectations from users, non-relational/unstructured data, and cloud-born data.  How can you prevent this from happening?  Enter the modern data warehouse, which is able to handle and excel with these new trends.  It handles all types of data (Hadoop), provides a way to easily interface with all these types of data (PolyBase), and can handle “big data” and provide fast queries.  Is there one appliance that can support this modern data warehouse?  Yes!  It is the Parallel Data Warehouse (PDW) from Microsoft, which is a Massively Parallel Processing (MPP) appliance that has been recently updated (v2 AU1).  In this session I will dig into the details of the modern data warehouse and PDW.  I will give an overview of the PDW hardware and software architecture, identify what makes PDW different, and demonstrate the increased performance.  In addition I will discuss how Hadoop, HDInsight, and PolyBase fit into this new modern data warehouse.

Posted in Data warehouse, Presentation, Session, SQLServerPedia Syndication | Leave a comment