Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Lee, Kyungbok, Paik, Myunghee Cho
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

_version_	1866929300294336512
author	Lee, Kyungbok Paik, Myunghee Cho
author_facet	Lee, Kyungbok Paik, Myunghee Cho
contents	We introduce a novel doubly-robust (DR) off-policy evaluation (OPE) estimator for Markov decision processes, DRUnknown, designed for situations where both the logging policy and the value function are unknown. The proposed estimator initially estimates the logging policy and then estimates the value function model by minimizing the asymptotic variance of the estimator while considering the estimating effect of the logging policy. When the logging policy model is correctly specified, DRUnknown achieves the smallest asymptotic variance within the class containing existing OPE estimators. When the value function model is also correctly specified, DRUnknown is optimal as its asymptotic variance reaches the semiparametric lower bound. We present experimental results conducted in contextual bandits and reinforcement learning to compare the performance of DRUnknown with that of existing methods.
format	Preprint
id	arxiv_https___arxiv_org_abs_2404_01830
institution	arXiv
publishDate	2024
record_format	arxiv
spellingShingle	Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy Lee, Kyungbok Paik, Myunghee Cho Machine Learning We introduce a novel doubly-robust (DR) off-policy evaluation (OPE) estimator for Markov decision processes, DRUnknown, designed for situations where both the logging policy and the value function are unknown. The proposed estimator initially estimates the logging policy and then estimates the value function model by minimizing the asymptotic variance of the estimator while considering the estimating effect of the logging policy. When the logging policy model is correctly specified, DRUnknown achieves the smallest asymptotic variance within the class containing existing OPE estimators. When the value function model is also correctly specified, DRUnknown is optimal as its asymptotic variance reaches the semiparametric lower bound. We present experimental results conducted in contextual bandits and reinforcement learning to compare the performance of DRUnknown with that of existing methods.
title	Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy
topic	Machine Learning
url	https://arxiv.org/abs/2404.01830

if (filemtime($filename) + $maxlifetime < time()) { unlink($filename); $count++; } } return $count; } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ protected function saveSession($sessId, $data): bool { $sessFile = $this->path . '/sess_' . $sessId; if ($handle = fopen($sessFile, 'w')) { $return = false; // Lock the file for exclusive access to avoid issues with multiple // processes writing session simultaneously: if (flock($handle, LOCK_EX)) { $return = fwrite($handle, $data); // Make sure that there's no trailing data by truncating the file to // the correct length: ftruncate($handle, strlen($data)); fflush($handle); flock($handle, LOCK_UN); } fclose($handle); if ($return !== false) { return true; } } // If we got this far, something went wrong with the file output... // It is tempting to throw an exception here, but this code is called // outside of the context of exception handling, so all we can do is // echo a message.

// Anecdotal testing Today and Yesterday seems to indicate destroy() // is called by the garbage collector and everything is good. // Something to keep in mind though. return 0; } /** * Write function that is called when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ public function write($sessId, $data): bool { if ($this->writesDisabled) { return true; } return $this->saveSession($sessId, $data); } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ abstract protected function saveSession($sessId, $data): bool; }

* * @return void */ public function writeClose() { // The assumption is that we're using PHP's ext/session. // session_write_close() will actually overwrite $_SESSION with an // empty array on completion -- which leads to a mismatch between what // is in the storage object and $_SESSION. To get around this, we // temporarily reset $_SESSION to an array, and then re-link it to // the storage object. // // Additionally, while you _can_ write to $_SESSION following a // session_write_close() operation, no changes made to it will be // flushed to the session handler. As such, we now mark the storage // object isImmutable. $storage = $this->getStorage(); if (! $storage->isImmutable()) { $_SESSION = $storage->toArray(true); session_write_close(); $storage->fromArray($_SESSION); $storage->markImmutable(); } } /** * Attempt to set the session name * * If the session has already been started, or if the name provided fails * validation, an exception will be raised. * * @param string $name * @return SessionManager * @throws Exception\InvalidArgumentException */ public function setName($name) { if ($this->sessionExists()) { throw new Exception\InvalidArgumentException( 'Cannot set session name after a session has already started'

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
VUFIND_SESSION	"8e9c674bfa6c224f50f0861b8975c5e8"
ui	"standard"
language	"fr"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1790882302.3469 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "8e9c674bfa6c224f50f0861b8975c5e8" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1433}
SessionHelper	Laminas\Stdlib\ArrayObject {#1762}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"VUFIND_SESSION=8e9c674bfa6c224f50f0861b8975c5e8; ui=standard; language=fr"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.142"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"34991"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2404_01830/Details """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2404_01830/Details"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1790882302.3425
REQUEST_TIME	1790882302

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
VUFIND_SESSION	"8e9c674bfa6c224f50f0861b8975c5e8"
ui	"standard"
language	"fr"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1790882302.3469 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "8e9c674bfa6c224f50f0861b8975c5e8" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1433}
SessionHelper	Laminas\Stdlib\ArrayObject {#1762}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"VUFIND_SESSION=8e9c674bfa6c224f50f0861b8975c5e8; ui=standard; language=fr"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.142"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"34991"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2404_01830/Details """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2404_01830/Details"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1790882302.3425
REQUEST_TIME	1790882302

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Documents similaires

Environment & details:

Environment & details: