Part Two: A Review of Microsoft's 2020 Carbon Negative accomplishments

by Contributed | Feb 23, 2021 | Technology

This article is contributed. See the original author and article here.

Carbon Negative

On January 28, 2021 Microsoft released its first Environmental Sustainability Report “A Year of Action” documenting progress the company made since announcing its 2020 commitments to become a carbon negative, water positive, zero waste company that protects more land than it uses by 2030. See a summary of the report in my previous blog post “Part one: A review of the Microsoft 2020 Environmental Sustainability Report A Year of Action.” This post will explore the carbon negative commitment. The third and final post will close out with a deeper look at water positive, zero waste and ecosystems, including the mysterious Planetary Computer.

Under construction: Climeworks’ new large-scale direct air capture and storage plant, “Orca.” (Credit: Climeworks)

Background

A company is carbon negative when it removes more carbon from the atmosphere than it emits each year. For the purposes of this post, carbon is shorthand for all greenhouse gases. If you want to understand why adding too much carbon to the atmosphere is bad, check out this great primer.

Microsoft will achieve carbon negativity through a combination of reduction and removal, making deep reductions of its emissions first and foremost, then using removal only for the residual footprint. This is rare: there aren’t many carbon negative companies, which means achieving this commitment will require operating in uncharted territory. Indeed, the report and accompanying white paper, Microsoft carbon removal “Lessons from an early corporate purchase”, make this point repeatedly as they describe the frustrating gap between ambition and reality. Today, we lack the technology, markets and commonly accepted accounting standards and methodologies required for carbon removal on a scale big enough to make a difference. Microsoft is going first and hoping to blaze a trail through the wilderness that others can follow.

Arguably this is where Microsoft focused most of its year one efforts, and for good reason. I recently had the chance to ask Microsoft’s Chief Environmental Officer Lucas Joppa “What keeps you up at night?” and the carbon problem was at the top of his list. From melting ice caps, to rising sea levels, more frequent extreme weather events, drought, heat waves, mass extinctions and a looming global migration crisis, the carbon problem is really one of mass systemic destabilization – the kind that could topple civilization as we know it. If climate science is telling us one thing most emphatically, it is that we need to drastically reduce the amount of carbon in the atmosphere immediately. This is why reduction of emissions simply will not be enough, for Microsoft or anyone else. As the chart below demonstrates, carbon removal will be critical to stabilizing the Earth’s climate.

Avoiding worst case climate scenarios will require removing emissions from the atmosphere.

Scoping it out

To learn more about Scope 1, 2, and 3 emissions, watch this 5 min video narrated by Lucas Joppa

Each of the 2020 carbon negative accomplishments represents an impressive amount of hard work and groundbreaking firsts. The complexity, challenge and scale of what went into the accomplishments cannot be overstated. What we see here is a company willing to go first and bring others along. Consider just a few of these points:

• Microsoft charges its own internal business groups a real fee on their emissions as an incentive to drive them down and a way to raise capital for funds that reward sustainable innovation internally. I am not aware of many companies, especially the size of Microsoft, who actually charge a real carbon fee – most use what are called “shadow prices.”


• Many companies haven’t yet begun to even measure their Scope 1 or 2 emissions, let alone reduce or remove them. Even fewer have done this for Scope 3


• Further, the company extended this real fee from its Scope 1 and 2 emissions out to Scope 3, which are the indirect emissions associated with its supply and value chain. This means it collects revenue on emissions it did not directly produce! Microsoft’s updated Supplier Code of Conduct essentially says “if you want to continue doing business with Microsoft, you will need to measure and disclose your own emissions because ultimately, they are part of our footprint and we’re taking responsibility for that.” In essence, the company is using a carrot and stick approach with its own internal business groups and its suppliers to ensure everyone is coming along on this carbon negative journey.

Why is the inclusion of Scope 3 emissions so significant? Look no further than this chart:

Like most organizations, Scope 3 accounts for the vast majority of Microsoft’s carbon emissions.

A whopping ~95% of Microsoft’s emissions are Scope 3 – which means that the company can only control 5% of its own carbon footprint directly. The vast majority of emissions comes from the upstream parties Microsoft relies on to build its products and eventually, the downstream parties that consume them. Simply put, driving down the emissions of Microsoft’s entire supply and value chain is the only viable path to carbon negativity by 2030 and that is no easy task. It’s a fascinating example of just how far the private sector can go in incentivizing behavior change in others. Actions like these are not a substitute for government action (like a long overdue price on carbon) but they are an ambitious and positive step in the right direction.

Drawdown

While these efforts are laudable they are simply not enough. Carbon emissions are measured in billions of metric tons – millions barely move the needle. Microsoft was only able to procure 1.3M metric tons of carbon removal in its first year (the largest annual purchase of carbon removal ever made by a single company). But this represents a tiny fraction – just 0.5% – of the 50B metric tons humanity pumps into the atmosphere every year. In order to sequester carbon at scale, Microsoft must help catalyze a market for carbon reduction that barely exists while relying on technology that is largely unproven.

A deeper look at Microsoft’s FY21 Carbon Removal Portfolio

“For the world to achieve a net-zero carbon economy, Microsoft’s removal demands must make up a tiny fraction global demand and supply – not the entirety of it.” – Lucas Joppa

 And that is largely the story of year one on Microsoft’s carbon negative journey. In my own words, what we need to accomplish carbon negativity doesn’t yet exist, but we won’t let that stop us from trying. We will deploy all our resources (money, technology, influence) to accelerate change and help others build what is needed. And we will be transparent about our successes and failures so others can follow suit, because frankly, the stakes could not be any higher.

“Stabilizing the global climate system will require a heroic societal effort. The world must drastically reduce carbon dioxide and other greenhouse gas (GHG) emissions. But reductions won’t be enough. As a global society we must also remove large amounts of carbon from the atmosphere, to avert the worst social, economic and environmental impacts of a rapidly changing climate. And we must do so while recovering from a pandemic.” – Carbon Program Manager Elizabeth Wilmott and Lucas Joppa

Want to go deeper and learn more about carbon removal?

• Microsoft carbon removal “Lessons from an early corporate purchase” white paper (read it!)


• Companion blog published by Microsoft’s Chief Environmental Officer Lucas Joppa


• Companion blog published by Microsoft’s President Brad Smith


• Explainer: Understanding the Math Behind Microsoft’s Commitment to Become Carbon Negative by 2030 – YouTube


• National Academies of Sciences, Engineering, and Medicine 2019 report


• 2021 Carbon Dioxide Removal Primer

Comments, questions and feedback welcome below! 

Drew

Jupyter Notebook Pivot Functions

by Contributed | Feb 23, 2021 | Technology

This article is contributed. See the original author and article here.

We recently released a new version of MSTICPy with a feature called Pivot functions.

This feature has three main goals:

• Making it easy to discover and invoke MSTICPy functionality.


• Creating a standardized way to call pivotable functions.


• Letting you assemble multiple functions into re-usable pipelines.

The pivot functionality exposes operations relevant to a particular entity as methods (or functions) of that entity. These operations include:

• Data queries


• Threat intelligence lookups


• Other data lookups such as geo-location or domain resolution


• and other local functionality

Here are a couple of examples showing calling different kinds of enrichment functions from the IpAddress entity:

>>> from msticpy.datamodel.entities import IpAddress, Host    
>>> IpAddress.util.ip_type(ip_str=“157.53.1.1”))
    ip          result    157.53.1.1  Public     
>>> IpAddress.util.whois(“157.53.1.1”))
    asn  asn_cidr  asn_country_code  asn_date    asn_description  asn_registry  nets …..    
        NA   NA        US                2015–04–01  NA               arin          [{‘cidr’: ‘157.53.0.0/16’…     >>> IpAddress.util.geoloc_mm(value=“157.53.1.1”))    CountryCode  CountryName    State   City   Longitude   Latitude   Asn…    US           United States  None    None   –97.822     37.751     None…

This second example shows a pivot function that does a data query for host logon events from a Host entity.

>>> Host.AzureSentinel.list_host_logons(host_name=“VictimPc”)
    Account               EventID   TimeGenerated                      Computer                 SubjectUserName   SubjectDomainName
    NT AUTHORITYSYSTEM  4624     2020–10–01 22:39:36.987000+00:00     VictimPc.Contoso.Azure      VictimPc$          CONTOSO

You can also add other functions from 3rd party Python packages or ones you write yourself as pivot functions.

Terminology

Before we get into things let’s clear up a few terms.

Entities – These are Python classes that represent real-world objects commonly encountered in CyberSec investigations and hunting. E.g., Host, URL, IP Address, Account, etc.

Pivoting – This comes from the common practice in CyberSec investigations of navigating from one suspect entity to another. E.g., you might start with an alert identifying a potentially malicious IP Address, from there you ‘pivot’ to see which hosts or accounts were communicating with that address. From there you might pivot again to look at processes running on the host or Office activity for the account.

Background reading

This article is available in Notebook form so that you can try out the examples. [TODO]

There is also full documentation of the Pivot functionality on our ReadtheDocs page.

Life before pivot functions

Before Pivot functions your ability to use the various bits of functionality in MSTICPy was always bounded by your knowledge of where a certain function was (or your enthusiasm for reading the docs).

For example, suppose you had an IP address that you wanted to do some simple enrichment on.

ip_addr = “20.72.193.242”

First, you’d need to locate and import the functions. There might also be (as in the GeoIPLiteLookup class) some initialization step you’d need to do before using the functionality.

from msticpy.sectools.ip_utils import get_ip_type
from msticpy.sectools.ip_utils import get_whois_info
from msticpy.sectools.geoip import GeoLiteLookup
geoip = GeoLiteLookup()

Next you might have to check the help for each function to work it parameters.

>>> help(get_ip_type)
Help on function get_ip_type in module msticpy.sectools.ip_utils: 
get_ip_type(ip: str = None, ip_str: str = None) -> str    
    Validate value is an IP address and deteremine IPType category.        … 

Then finally run the functions.

>>> get_ip_type(ip_addr)
‘Public’

>>> get_whois_info(ip_addr)
(‘MICROSOFT-CORP-MSN-AS-BLOCK, US’, 
    {‘nir’: None,  ‘asn_registry’: ‘arin’,  ‘asn’: ‘8075’,  ‘asn_cidr’: ‘20.64.0.0/10’,  ‘asn_country_code’: ‘US’,  ‘asn_date’: ‘2017-10-18’,  ‘asn_description’: ‘MICROSOFT-CORP-MSN-AS-BLOCK, US’,  ‘query’: ‘20.72.193.242’,  ‘nets’: …

>>> geoip.lookup_ip(ip_addr)
([{‘continent’: 
    {‘code’: ‘NA’,    ‘geoname_id’: 6255149,    ‘names’: 
       {‘de’: ‘Nordamerika’,     ‘en’: ‘North America’,     ‘es’: ‘Norteamérica’,     ‘fr’: ‘Amérique du Nord’,     ‘ja’: ‘北アメリカ’,     ‘pt-BR’: ‘América do Norte’,     ‘ru’: ‘Северная Америка’,     ‘zh-CN’: ‘北美洲’}},   ‘country’: {‘geoname_id’: 6252001,    ‘iso_code’: ‘US’,    ‘names’: {‘de’: ‘USA’,     ‘en’: ‘United States’,     ‘es’: ‘Estados Unidos’,     …

At which point you’d discover that the output from each function was somewhat raw and it would take a bit more work if you wanted to combine it in any way (say in a single table).

In the rest of the article we’ll show you how Pivot functions make it easier to discover data and enrichment functions. We’ll also show how pivot functions bring standardization and handle different types of input (including lists and DataFrames) and finally, how the standardized output lets you chain multiple pivot functions together into re-usable pipelines of functionality.

Getting started with pivot functions

Let’s get started with how to use Pivot functions.

Typically, we use MSTICPy‘s init_notebook function at the start of any notebook. This handles checking versions and importing some commonly-used packages and modules (both MSTICPy and 3rd party packages like pandas).

>>> from msticpy.nbtools.nbinit import init_notebook
>>> init_notebook(namespace=globals());
Processing imports……..

Then there are a couple of preliminary steps needed before you can use pivot functions. The main one is loading the Pivot class.

Pivot functions are added to the entities dynamically by the Pivot class. The Pivot class will try to discover relevant functions from queries, Threat Intel providers and various utility functions.

In some cases, notably data queries, the data query functions are themselves created dynamically, so these need to be loaded before you create the Pivot class. (You can always create a new instance of this class, which forces re-discovery, so don’t worry if mess up the order of things).

Note in most cases we don’t need to connect/authenticate to a data provider prior to loading Pivot.

Let’s load our data query provider for AzureSentinel.

>>> az_provider = QueryProvider(“AzureSentinel”)
Please wait. Loading Kqlmagic extension…

Now we can load and instantiate the Pivot class.

>>> from msticpy.datamodel.pivot import Pivot
>>> pivot = Pivot(namespace=globals())

Why do we need to pass “namespace=globals()”? 

Pivot searches through the current objects defined in the Python/notebook namespace to find provider objects that it will use to create the pivot functions. This is most relevant for QueryProviders – when you create a Pivot class instance it will find and use the relevant queries from the az_provider object that we created in the previous step. In most other cases (like GeoIP and ThreatIntel providers, it will create new ones if it can’t find existing ones).

Easy discovery of functionality

Find the entity name you need

The simplest way to do this is simply enumerate (using Python dir() function) the contents of the MSTICPy entities sub-package. This should have already been imported by the init_notebook function that we ran earlier.

The items at the beginning of the list with proper capitalization are the entities.

>>> dir(entities)
[‘Account’, ‘Alert’, ‘Algorithm’, ‘AzureResource’, ‘CloudApplication’, ‘Dns’, ‘ElevationToken’, ‘Entity’, ‘File’, ‘FileHash’, ‘GeoLocation’, ‘Host’, ‘HostLogonSession’, ‘IpAddress’, ‘Malware’, …

We’re going to make this a little more elegant in a forthcoming update with this helper function.

>>> entities.find_entity(“ip”)
Match found ‘IpAddress’msticpy.datamodel.entities.ip_address.IpAddress

Listing pivot functions available for an entity

Note you can always address an entity using its qualified path, e.g. “entities.IpAddress” but if you are going to use one or two entities a lot, it will save a bit of typing if you import them explicitly.

>>> from msticpy.datamodel.entities import IpAddress, Host

Once you have the entity loaded, you can use the get_pivot_list() function to see which pivot functions are available for it. The example below has been abbreviated for space reasons.

>>> IpAddress.get_pivot_list()
[‘AzureSentinel.SecurityAlert_list_alerts_for_ip’,
‘AzureSentinel.SigninLogs_list_aad_signins_for_ip’,
‘AzureSentinel.AzureActivity_list_azure_activity_for_ip’,
‘AzureSentinel.AzureNetworkAnalytics_CL_list_azure_network_flows_by_ip’,
…
‘ti.lookup_ip’,
‘ti.lookup_ipv4’,
‘ti.lookup_ipv4_OTX’,
…
‘ti.lookup_ipv6_OTX’, 
‘util.whois’, 
‘util.ip_type’, 
‘util.ip_rev_resolve’, 
‘util.geoloc_mm’, 
‘util.geoloc_ips’]

Some of the function names are a little unwieldy but, in many cases, this is necessary to avoid name collisions. You will notice from the list that the functions are grouped into containers: “AzureSentinel”, “ti” and “util” in the above example.

Although this makes the function name even longer, we thought that this helped to keep related functionality together – so you don’t get a TI lookup function, when you thought you were running a query.

Fortunately, Jupyter notebooks/IPython support tab completion so you should not normally have to remember these names.

The containers (“AzureSentinel”, “util”, etc.) are also callable functions – they just return the list of functions they contain.

>>> IpAddress.util()
whois functionip_type functionip_rev_resolve functiongeoloc_mm functiongeoloc_ips function

Now we’re ready to run any of the functions for this entity (we take the same initial examples from the “Life before pivot functions” plus a few more).

>>> IpAddress.util.ip_type(ip_addr)

>>> IpAddress.util.whois(ip_addr)

>>> IpAddress.util.ip_rev_resolve(ip_addr)

>>> IpAddress.util.geoloc_mm(ip_addr)

>>> IpAddress.ti.lookup_ip(ip_addr)

Notice that we didn’t need to worry about either the parameter name or format (more on this in the next section). Also, whatever the function, the output is always returned as a pandas DataFrame.

For Data query functions you do need to worry about the parameter name

Data query functions are slightly more complex than most other functions and specifically often support many parameters. Rather than try to guess which parameter you meant, we require you to be explicit about it.

Before we can use a data query, we need to authenticate to the provider.

>>> az_provider.connect(WorkspaceConfig(workspace=“CyberSecuritySoc”).code_connect_str)

If you are not sure of the parameters required by the query you can use the built-in help

>>> Host.AzureSentinel.SecurityAlert_list_related_alerts?
Signature: Host.AzureSentinel.SecurityAlert_list_related_alerts(*args, **kwargs) -> Union[pandas.core.frame.DataFrame, Any]
Docstring:
Retrieves list of alerts with a common host, account or process
Parameters
———-
account_name: str (optional)
    The account name to find
add_query_items: str (optional)
    Additional query clauses
end: datetime (optional)
    Query end time
host_name: str (optional)
    The hostname to find
path_separator: str (optional)
    Path separator    (default value is: )
process_name: str (optional) …

In this case we want the “host_name” parameter.

>>> Host.AzureSentinel.SecurityAlert_list_related_alerts(host_name=“victim00”).head(5)

Shown below is a preview of a notebook tool that lets you browser around entities and their pivot functions, search for a function by keyword and view the help for that function. This is going to be released shortly.

>>> Pivot.browse()

Standardized way of calling Pivot functions

Due to various factors (historical, underlying data, developer laziness and forgetfulness, etc.) the functionality in MSTICPy can be inconsistent in the way it uses input parameters.

Also, many functions will only accept inputs as a single value, or a list or a DataFrame or some unpredictable combination of these.

Pivot functions allow you to largely forget about this – you can use the same function whether you have:

• a single value


• a list of values (or any Python iterable, such as a tuple or even a generator function) 


• a DataFrame with the input value in one of the columns.

Let’s take an example.

Suppose we have a set of IP addresses pasted from somewhere that we want to use as input.

We need to convert this into a Python data object of some sort.

To do this we can use another Pivot utility %%txt2df. This is a Jupyter/IPython magic function – to use it, just paste you data in a cell that you want to import into an empty. Use

 %%txt2df –help

in an empty cell to see the full syntax.

In the example below, we specify a comma separator, that the data has a headers row and to save the converted data as a DataFrame named “ip_df”.

Warning if you specify the “–name” parameter, this will overwrite any existing variable of this name.

%%txt2df –sep , –headers –name ip_df
idx, ip, type
0, 172.217.15.99, Public
1, 40.85.232.64, Public
2, 20.38.98.100, Public
3, 23.96.64.84, Public
4, 65.55.44.108, Public
5, 131.107.147.209, Public
6, 10.0.3.4, Private
7, 10.0.3.5, Private
8, 13.82.152.48, Public

For demonstration purposes, we’ll also create a standard Python list from the “ip” column of the DataFrame.

>>> ip_list = list(ip_df.ip)
>>> print(ip_list)
[‘172.217.15.99’, ‘40.85.232.64’, ‘20.38.98.100’, ‘23.96.64.84’, ‘65.55.44.108’, ‘131.107.147.209’, ‘10.0.3.4’, ‘10.0.3.5’, ‘13.82.152.48’]

How did this work before?

If you recall the earlier example of get_ip_type, passing it a list or DataFrame doesn’t result in anything useful.

>>> get_ip_type(ip_list)
[‘172.217.15.99’, ‘40.85.232.64’, ‘20.38.98.100’, ‘23.96.64.84’, ‘65.55.44.108’, ‘131.107.147.209’, ‘10.0.3.4’, ‘10.0.3.5’, ‘13.82.152.48’]
does not appear to be an IPv4 or IPv6 address
‘Unspecified’

Pivot versions are (somewhat) agnostic to input data format

However, the “pivotized” version can accept and correctly process a list.

>>> IpAddress.util.ip_type(ip_list)

In the case of a DataFrame, we have to tell the function the name of the column that contains the input data.

>>> IpAddress.util.whois(ip_df)  # won’t work!
—————————————————————————
KeyError                                  Traceback (most recent call last)
<ipython-input-32-debf57d805c7> in <module>
…
    173         input_df, input_column, param_dict = _create_input_df(
–> 174             input_value, pivot_reg, parent_kwargs=kwargs
    175         )
…
KeyError: (“‘ip_column’ is not in the input dataframe”, 
‘Please specify the column when calling the function.
You can use one of the parameter names for this:’, [‘column’, ‘input_column’, ‘input_col’, ‘src_column’, ‘src_col’])

>>> entities.IpAddress.util.whois(ip_df, column=”ip”)  # correct

Note: for most functions you can ignore the parameter name and just specify it as a positional parameter. You can also use the original parameter name of the underlying function or the placeholder name “value”.

The following are all equivalent:

>>> IpAddress.util.ip_type(ip_list)
>>> IpAddress.util.ip_type(ip_str=ip_list)
>>> IpAddress.util.ip_type(value=ip_list)
>>> IpAddress.util.ip_type(data=ip_list)

When passing both a DataFrame and column name use:

>>> IpAddress.util.ip_type(data=ip_df, column=“col_name”)

You can also pass an entity instance of an entity as a input parameter. The pivot code knows which attribute or attributes of an entity will provider the input value.

>>> ip_entity = IpAddress(Address=”40.85.232.64″)
>>> IpAddress.util.ip_type(ip_entity)

Iterable/DataFrame inputs and single-value functions

Many of the underlying functions only accept single values as inputs. Examples of these are the data query functions – typically they expect a single host name, IP address, etc.

Pivot knows about the type of parameters that the function accepts. It will adjust the input to match the expectations of the underlying function. If a list or DataFrame is passed as input to a single-value function Pivot will split the input and call the function once for each value. It then combines the output into a single DataFrame before returning the results.

You can read a bit more about how this is done in the Appendix – “how do pivot wrappers work?”

Data queries – where does the time range come from?

The Pivot class has a built-in time range, which is used by default for all queries. Don’t worry – you can change it easily.

To see the current time setting:

>>> Pivot.current.timespan
TimeStamp(start=2021-02-15 21:01:40.381864, end=2021-02-16 21:01:40.381864, period=-1 day, 0:00:00)

Note: “Pivot.current” gives you access to the last created instance of the Pivot class – if you’ve created multiple instances of Pivot (which you rarely need to do), you can always get to the last one you created using this class attribute.

You can edit the time range interactively

Pivot.current.edit_query_time()

Or by setting the timespan property directly.

>>> from msticpy.common.timespan import TimeSpan
>>> # TimeSpan accepts datetimes or datestrings
>>> timespan = TimeSpan(start=”02/01/2021″, end=”02/15/2021″)
>>> Pivot.current.timespan = timespan
TimeStamp(start=2021-02-01 00:00:00, end=2021-02-15 00:00:00, period=-14 days +00:00:00)

In an upcoming release there is also a convenience function for setting the time directly with Python datetimes or date strings.

Pivot.current.set_timespan(start=”2020-02-06 03:00:00″, end=”2021-02-15 01:42:42″)

You can also override the built-in time settings by specifying start and end as parameters to the query function.

Host.AzureSentinel.SecurityAlert_list_related_alerts(host_name=”victim00″, start=dt1, end=dt2)

Supplying extra parameters

The Pivot layer will pass any unused keyword parameters to the underlying function. This does not usually apply to positional parameters – if you want parameters to get to the function, you have to name them explicitly. In this example the add_query_items parameter is passed to the underlying query function

>>> entities.Host.AzureSentinel.SecurityEvent_list_host_logons(
        host_name=”victimPc”,
        add_query_items=”| summarize count() by LogonType”
    )

Pivot Pipelines

Because all pivot functions accept DataFrames as input and produce DataFrames as output, it means that it is possible to chain pivot functions into a pipeline.

Joining input to output

You can join the input to the output. This usually only makes sense when the input is a DataFrame. It lets you keep the previously accumulated results and tag on the additional columns produced by the pivot function you are calling.

The join parameter supports “inner”, “left”, “right” and “outer” joins (be careful with the latter though!) See pivot joins documentation for more details.

Although joining is useful in pipelines you can use it on any function whether in a pipeline or not. In this example you can see that the idx, ip and type columns have been carried over from the source DataFrame and joined with the output.

>>> entities.IpAddress.util.whois(ip_df, column=”ip”, join=”inner”)

Pipelines

Pivot pipelines are implemented pandas customr accessors. Read more about Extending pandas here.

When you load Pivot it adds the mp_pivot pandas DataFrame accessor. This appears as an attribute to DataFrames.

>>> ips_df.mp_pivot
<msticpy.datamodel.pivot_pd_accessor.PivotAccessor at 0x275754e2208>

The main pipelining function run is a method of mp_pivot. run requires two parameters

• the pivot function to run and


• the column to use as input.

See mp_pivot.run documentation for more details.

Here is an example of using it to call four pivot functions, each using the output of the previous function as input and using the join parameter to accumulate the results from each stage.

1. (
2.     ips_df
3.     .mp_pivot.run(IpAddress.util.ip_type, column=”IP”, join=”inner”)
4.     .query(“result == ‘Public'”).head(10)
5.     .mp_pivot.run(IpAddress.util.whois, column=”ip”, join=”left”)
6.     .mp_pivot.run(IpAddress.util.geoloc_mm, column=”ip”, join=”left”)
7.     .mp_pivot.run(IpAddress.AzureSentinel.SecurityAlert_list_alerts_for_ip, source_ip_list=”ip”, join=”left”)
8. ).head(5)

Let’s step through it line by line.

1. The whole thing is surrounded by a pair of parentheses – this is just to let us split the whole expression over multiple lines without Python complaining.


2. Next we have ips_df – this is just the starting DataFrame, our input data.


3. Next we call the mp_pivot.run() accessor method on this dataframe. We pass it the pivot function that we want to run (IpAddress.util.ip_type) and the input column name (IP). This column name is the column in ips_df where our input IP addresses are. We’ve also specified an join type of “inner”. In this case the join type doesn’t really matter since we know we get exactly one output row for every input row.


4. We’re using the pandas query function to filter out unwanted entries from the previous stage. In this case we only want “Public” IP addresses. This illustrates that you can intersperse standard pandas functions in the same pipeline. We could have also added a column selector expression ([[“col1”, “col2″…]]), for example, if we wanted to filter the columns passed to the next stage


5. We are calling a further pivot function – whois. Remember the “column” parameter always refers to the input column, i.e. the column from previous stage that we want to use in this stage.


6. We are calling geoloc_mm to get geo location details joining with a “left” join – this preserves the input data rows and adds null columns in any cases where the pivot function returned no result.


7. Is the same as 6 except the called function is a data query to see if we have any alerts that contain these IP addresses. Remember, in the case of data queries we have to name the specific query parameter that we want the input to go to. In this case, each row value in the ip column from the previous stage will be sent to the query.


8. Finally we close the parentheses to form a valid Python expression. The whole expression returns a DataFrame so we can add further pandas operations here (like .head(5) shown here).

>>> (
       ips_df
       .mp_pivot.run(entities.IpAddress.util.ip_type, column=”IP”, join=”inner”)
       .query(“result == ‘Public'”).head(10)
       .mp_pivot.run(entities.IpAddress.util.whois, column=”ip”, join=”left”)
       .mp_pivot.run(entities.IpAddress.util.geoloc_mm, column=”ip”, join=”left”)
       .mp_pivot.run(entities.IpAddress.AzureSentinel.SecurityAlert_list_alerts_for_ip, source_ip_list=”ip”, join=”left”)
   ).head(5)

Other pipeline functions

In addition to run, the mp_pivot accessor also has the following functions:

• display – this simply displays the data at the point called in the pipeline. You can add an optional title, filtering and the number or rows to display


• tee – this forks a copy of the DataFrame at the point it is called in the pipeline. It will assign the forked copy to the name given in the var_name parameter. If there is an existing variable of the same name it will not overwrite it unless you add the clobber=True parameter.

In both cases the pipelined data is passed through unchanged.

See Pivot functions help for more details.

Use of these is shown below in this partial pipeline.

    …
    .mp_pivot.run(IpAddress.util.geoloc_mm, column=”ip”, join=”left”)
    .mp_pivot.display(title=”Geo Lookup”, cols=[“IP”, “City”])  # << display an intermediate result
    .mp_pivot.tee(var_name=”geoip_df”, clobber=True)  # << save a copy called ‘geoip_df’
    .mp_pivot.run(IpAddress.AzureSentinel.SecurityAlert_list_alerts_for_ip, source_ip_list=”ip”, join=”left”)

In the next release we’ve also implemented:

• tee_exec – this executes a function on a forked copy of the DataFrame The function must be a pandas function or custom accessor. A good example of the use of this might be creating a plot or summary table to display partway through the pipeline.

Extending Pivot – adding your own (or someone else’s) functions

You can add pivot functions of your own. You need to supply:

• the function


• some metadata that describes where the function can be found and how the function works

Full details of this are described here.

The current version of Pivot doesn’t let you add functions defined inline (i.e. written in the notebook itself) but this will be possible in the forthcoming release.

Let’s create a function in a Python module my_module.py. We can do this using the %%write_file magic function and running the cell.

%%writefile my_module.py
“””Upper-case and hash”””
from hashlib import md5

def my_func(input: str):
    md5_hash = “-“.join(hex(b)[2:] for b in md5(input.encode(“utf-8”)).digest())
    return {
        “Title”: input.upper(),
        “Hash”: md5_hash
    }

We also need to create a YAML definition file for our pivot function. Again we can use %%write_file to create a local file in the current directory. We need to tell Pivot

• the name of the function and source module,


• the name of the container that the function will appear in,


• the input type expected by the function (“value”, “list” or “dataframe”)


• which entities to add the pivot to, along with a corresponding attribute of the entity. (The attribute is used in cases where you are passing an instance of an entity itself as an input parameter – if in doubt just use any valid attribute of the entity).


• The name of the input attribute of the underlying function.


• An (optional) new name to give the function.

%%writefile my_func.yml
pivot_providers:
  my_func_defn:
    src_func_name: my_func
    src_module: my_module
    entity_container_name: cyber
    input_type: value
    entity_map:
      Host: HostName
    func_input_value_arg: input
    func_new_name: upper_hash_name

Now we can register the function we created as a pivot function.

>>> from msticpy.datamodel.pivot_register_reader import register_pivots
>>> register_pivots(“my_func.yml”)

An then run it.

>>> Host.cyber.upper_hash_name(“host_name”)

In the next release, this will be available as a simple function that can be used to add a function defined in the notebook as shown here.

from hashlib import md5

def my_func2(input: str):
    md5_hash = “-“.join(hex(b)[2:] for b in md5(input.encode(“utf-8”)).digest())
    return {
        “Title”: input.upper(),
        “Hash”: md5_hash
    }


Pivot.add_pivot_function(
    func=my_func2,
    container=”cyber”,  # which container it will appear in on the entity
    input_type=”value”,
    entity_map={“Host”: “HostName”},
    func_input_value_arg=”input”,
    func_new_name=”il_upper_hash_name”,
)

Host.cyber.il_upper_hash_name(“host_name”)

Conclusion

We’ve taken a short tour through the MSTICPy Pivot functions, looking at how they make the functionality in MSTICPy easier to discover and use.

I’m particularly excited about the pipeline functionality. In the next release we’re going to make it possible to define reusable pipelines in configuration files and execute them with a single function call. This should help streamline some common patterns in notebooks for Cyber hunting and investigation.

Please send any feedback or suggestions for improvements to msticpy@microsoft.com or create an issue on https://github.com/microsoft/msticpy.

Happy hunting!

Appendix – how do pivot wrappers work?

In Python you can create functions that return other functions. This is called wrapping the function.

It allows the outer function to do additional things to the input parameters and the return value of the inner function.

Take this simple function that just applies proper capitalization to an input string.

def print_me(arg):
    print(arg.capitalize())

print_me(“hello”)

Hello

If we try to pass a list to this function we get an expected exception since the function only knows how to process a string

print_me([“hello”, “world”])
—————————————————————————
AttributeError                            Traceback (most recent call last)
<ipython-input-36-94b3e61eb86f> in <module>
…
AttributeError: ‘list’ object has no attribute ‘capitalize’

We could create a wrapper function that checked the input and iterated over the individual items if arg is a list. The works but we don’t want to have to do this for every function that we want to have flexible input!

def print_me_list(arg):
    if isinstance(arg, list):
        for item in arg:
            print_me(item)
    else:
        print_me(arg)

print_me_list(“hello”)
print_me_list([“how”, “are”, “you”, “?”])

Hello

How

Are

You

?

Instead, we can create a function wrapper.

In the example below, the outer function dont_care_func defines an inner function – list_or_str – and then returns this function. The inner function list_or_str is what implements the same “is-this-a-string-or-list” logic that we saw in the previous example. Crucially though, it isn’t hard-coded to call print_me but calls whatever function is passed (the func parameter) to it from the outer function dont_care_func.

# Our magic wrapper
def dont_care_func(func):

    def list_or_str(arg):
        if isinstance(arg, list):
            for item in arg:
                func(item)
        else:
            func(arg)
    return list_or_str

How do we use this?

We simply pass the function that we want to wrap to dont_care_func. Recall, that this function just returns an instance of the inner function. In this case the value func will have been replaced by the actual function print_me.

 print_stuff = dont_care_func(print_me)

Now we have a wrapped version of print_me that can handle different types of input. Magic!

print_stuff(“hello”)
print_stuff([“how”, “are”, “you”, “?”])

Hello

How

Are

You

?

We can also define further functions and create wrapped versions of those by passing them to dont_care_func.

def shout_me(arg):
    print(arg.upper(), “U0001F92C!”, end=” “) 

shout_stuff = dont_care_func(shout_me)

shout_stuff(“hello”)
shout_stuff([“how”, “are”, “you”, “?”])

HELLO ?! HOW ?! ARE ?! YOU ?! ? ?!

The wrapper functionality in Pivot is a bit more complex than this but essentially operates this way.

The Adoption Hub: This MS Teams app just got itself some upgrades…(do you have the latest?)

by Contributed | Feb 23, 2021 | Technology

This article is contributed. See the original author and article here.

Time flies by so fast – An expression we all have used in many aspects of our lives. Time flies when you’re having fun. Do we have to go now? They sure do grow up quickly. As a father of one, with one on the way, I truly didn’t fully understand what this meant until I became a parent. Watching your children grow, making friends and seeing them develop into so much more…

…it simply goes by way too fast.

Then there is technology. From Windows 95 to Windows 10. From the first iPhone to iPhone 12. Technology is continuously being pushed beyond it’s limits to deliver more than it what was originally developed for.

Link for above article here: Boost your technology adoption efforts with this single MS Teams app | LinkedIn

Our MS Teams “Adoption App” (Boost your technology adoption efforts with this single MS Teams app) that was shared earlier this year, has received a lot of praise since it was just launched this past January. With praise, comes requests, comments and suggestions…and we wanted to let you all know….we heard you! So let’s get right down to it…

1. Name Change – We began to think the Adoption App (V1) was one that managed information within a single hub that was easily accessible. Which helped us to make the easy decision of renaming to Adoption Hub (V2).

2. Logo Change – Following the hub theme, we believe the new logo is a better representation of what the app actually does.

3. Bot – We sped up our idea to include a bot. It was one of the top top top conversations we had with our peers and our customers. Bots have so much to bring to the table when it comes to providing information out to end users.

4. GCC Friendly – The Adoption App (V1) is now Government Community Cloud (GCC) friendly. Working with fellow CSMs, we were able to add additional documentation to support the processes of a GCC tenant.

Ready to get started? Go to aka.ms/theadoptionhub for details and how to deploy.

Now it is time to for Aaron Porzondek and I to thank a select group of very special individuals. They have inspired us, supported us, coached us and have joined us.

Christian Howe – Honestly. Aaron and I wouldn’t be here at all if it weren’t for Christian taking the time to show us the “origin app” if you will. Which inspired us to learn more about this framework and build out what the Adoption Hub is today.

Na-Young Kwon and Jace Moreno – With your unwavering support and continued guidance, it has motivated us to discover and find challenges to push us further along our Teams as a Platform journey.

Matt Hickey and Erik Olsson – Bots. Bots. And more bots. Thank you both for educating and ramping us up on Power Virtual Agents (PVA) bots. Having a bot within the Adoption Hub will certainly help end users with the understanding of technologies within an organization, such as MS Teams.

Glenn Blinckmann and Jeff Hicks – What started out as a show-and-tell of the Adoption App (V1), who knew it would lead to a great partnership between two OUs (MS talk). With Glenn’s and Jeff’s knowledge of how things need to be achieved within GCC, we were able to add GCC friendly instructions into Adoption App (V1) deployment guides.

HLS MW CSMs – Your encouragement, your ideas, your contributions, have greatly influenced our thought processes and decisions to take the direction we did of “platformizing the MW CSM playbook”, to help our customers and our fellow CSMs.

And to show our gratitude to you all and to the many others who have supported us…

Better together: Accomplish more during your day with Teams + OneDrive

by Contributed | Feb 23, 2021 | Technology

This article is contributed. See the original author and article here.

For most people, remote work—whether part-time or full-time—is the new way of working. That’s why over 115 million daily users rely on Microsoft Teams to connect and collaborate, and also why Teams was recognized as a Leader in the Gartner Magic Quadrant for Unified Communications as a Service. We designed Teams, your digital hub for teamwork, and OneDrive, your personal cloud storage that lets you access your files from anywhere, to work seamlessly and securely together. You can use Microsoft 365 Apps including Word, PowerPoint, Excel, or OneNote notebooks to create your work, store those files using OneDrive, and share them securely with colleagues in Teams, where you can co-author documents, meet with your teammates, and chat or meet with people inside or outside organization, all in one place.

Imagine a day in the life using Teams + OneDrive

Let’s walk through a day of using OneDrive and Teams together for work. You’re preparing a PowerPoint presentation for a conference call with an important client, Contoso, later in the week. In the morning, you open Teams and select the Files list, which shows you Recent files you’ve been working on, your Microsoft Teams files, and Downloads, as well as any cloud storage services connected with Teams. OneDrive is the default files app for Microsoft 365. When you click on OneDrive, you can see all your individual files for work. You can quickly add, upload, and sync files right in Teams, or open in them directly OneDrive.

In the OneDrive view, you navigate to a folder called “Contoso Electronics” and open the Contoso Electronics Sales presentation draft you’ve been working on and recently shared with your manager for review via chat in Teams. When you share your OneDrive files in chat, each file has unique permissions granted based on the person or people in the chat—that way, you don’t have to worry about unintended recipients accessing your content.

After you’ve reviewed your manager’s edits and updated the presentation, you decide it’s ready for the whole team to review. Your department has a team called “ Contoso” where you share and manage all deliverables for the client. Whenever a team is created, a corresponding Microsoft 365 Group and SharePoint shared library are also automatically created. All the files created in or uploaded to your  Contoso folder are stored and backed up in the SharePoint library. You copy the presentation from your OneDrive to the documents folder within Contoso team and then select the Posts tab to leave a message to let everyone on your team know the file is ready for review. Leveraging OneDrive you can sync all the files in the Contoso shared library directly to your device so you can work on them while offline.

Due to the deep integration between OneDrive, Teams, and Office, your team can select from a variety of tools to annotate, highlight, and comment on content. They can use @mentions to flag comments and tasks for you or other reviewers, and they can also track version history and restore previous file versions as needed. And because all edits are synced and stored in the cloud, they can start editing a document on one device and finish it on another. When the afternoon team meeting to review the presentation begins, everyone can easily access the file  to review each other’s comments and use co-authoring to finish editing the presentation together in real time.

Later in the day, you’ve scheduled focus time to work on a project outlining the go-to market plan for a new product series. A co-worker has already started the outline and uploaded the document to the Go-To Market channel in Teams. You decide to work on the document at a park near your house, so you take your tablet. Since you had already added a shortcut to the Go To Market shared library using the Add to OneDrive feature, which brings all your shared content from OneDrive, Teams, and SharePoint into one place , you can swiftly fetch the outline and start working on the same right within your OneDrive app.  You also realize that several GTM deliverables are missing approved vendor contracts. Using the file request feature in OneDrive you create an upload-only link and share the same in your “Partner Program“ team which consists of all your suppliers as guests members. This enables each vendor to upload their proposed scope of work directly to your OneDrive in a location that you chose—without having visibility to the other files in the folder.

Share files and collaborate securely, with peace of mind

While popular online storage apps integrate with Teams so you can access and share files, OneDrive provides a more secure sharing experience to help control data leakage and access to sensitive company information. Having your content in OneDrive enables you to share files as a link, internally and externally, so that every recipient has access to the most updated version.  Depending on how your IT organization has configured sharing permissions in OneDrive and SharePoint, you can set permissions for who can access that link—anyone, people only inside the organization,  specific people or people in the Teams group chat—and whether they can view or edit the file. You can also set expiration dates (for example, for outside vendors you don’t want accessing files or folders past a certain date) or set passwords to protect sensitive company or employee information. Blocking downloads on files also prevents recipients from saving files to their computers.

Sharing integration in Microsoft Teams

Work confidently, knowing that IT can protect your data

Exposure of sensitive company or client information can have serious legal and compliance implications. Today’s remote working environment can heighten these worries for IT, because people need to share information outside the bounds of a protected company environment. Teams and OneDrive not only provide coherent collaborative experiences for you but also bring consistency for the admins as they maintain the productivity apps. Instead of managing multiple third-party tools in silo the seamless integration between Teams and OneDrive empower admins to set governance and compliance policies at an organization level that can be extended to both OneDrive and Teams.

To keep you protected, IT can configure secure sharing policies in  OneDrive which automatically gets adopted by Teams, ensuring you have the tools you need to collaborate securely and consistently.

IT can also use Microsoft Information Protection to create policies for automatic classification of sensitive data, so if you create a document that contains sensitive client data, that document will automatically be classified by the system and encrypted for additional protection. This takes the burden of worrying about security off you, letting you focus on getting work done. Using information barriers, IT can also restrict communication and collaboration in Teams or OneDrive between two departments or segments to avoid a conflict of interest from occurring or between certain people to safeguard internal information.

IT can also keep an eye on how you and your team interact with shared content, adding an extra layer of security and control. Through detailed audit logs and reports available in the Microsoft 365 Security and Compliance Center, IT can trace OneDrive activity at the folder, file, and user levels, so they can see at a glance if any unauthorized users have tried to access sensitive company or client information. Every user action, including changes and modifications made to files and folders, is recorded for a full audit trail. In addition, even remotely, IT has the device visibility and control that’s especially important for thwarting breaches and ransomware attacks.

To learn more about why Teams and OneDrive are better together and how you can streamline your workday, check out our latest episode on Sync Up- a OneDrive podcast where we talk with Cory Kincaid, a Customer Success Manager for Modern Work, who advises customers on how to use technologies like Teams and OneDrive to improve their business.   

https://html5-player.libsyn.com/embed/episode/id/18023576/height/90/theme/custom/thumbnail/yes/direction/backward/render-playlist/no/custom-color/f99400/

Thank you for your time reading all about OneDrive,

Ankita Kirti 

OneDrive